LifeGame

Fallen Angles

I normally don't like making predictions, but I'll make an exception. I predict that sometime in the next few months (call it by June or July) Obama's approval rating will drop significantly. I have no idea what will be blamed for the cause, but I do know that the event responsible will unlikely be related to the drop.

How can I make this kind of prediction? Because his popularity started high. It can go nowhere but down and so down it will go.

This is part of a much larger problem that has to do with expectation - a key problem in management of any project, games included. It's quite simple really. How "well" you are doing is based on the relative improvement over the last marked measurement point. It is not possible for us to accept that sometimes you will do well and sometimes you will do poorly without having some kind of reasons for "something going wrong." This can be seen everywhere where performance is measured: sports teams, crime fighting, the economy and projects. In business, in extreme cases, such companies are called "Fallen Angles."

How well things are going is based much more on our perception of that thing combined with its relative improvement or drop. When a company is taking risks is doing well it is called "bold" and "visionary." When it does poorly those same decisions are due to "recklessness" and "lack of focus." Don't believe me? Check out this article: http://cmr.berkeley.edu/articles/accenture_award_08/2007_49_4_5447.pdf or better yet read "The Halo Effect."

One of the early studies done divided students into groups to carry out a project. After the project was done, the students were graded on their group performance and then asked to rate group members in various categories. What the study discovered, to no one's surprise, was that a higher grade correlated to group members rating each other more highly. The twist is that the grades were assigned randomly. Thus, it is not the team member's abilities that were a predictor of how well they were rated, but rather, it was the outcome that was the cause of how members were rated. We don't like to believe that. We like to think that evaluate people based on their ability independent of the outcome of their decisions, but it just ain't so. Results dictate how we see people. This is why people who seem like saviors one day will be "screwing up" the next. And it's why Obama's popularity will go down.

Think of another experiment. What makes people happier: getting 1$ a day for 100 days or getting 100$ once in one day. Most people choose a. Now flip it around. What is worse, losing 1$ a day for 100 days or losing 100$ once. Most people choose B. This is because there is an emotional overhead per event. In case A, the impact is positive, in case B it is negative.

This is important for managers to understand. Not only are you going to be evaluated on how much better or worse you are doing (as opposed to how well or poorly you are doing), but you will also evaluate people based on that same criteria.

Internally we use scrum to track projects. This is great and also picking up popularity. There is one major problem, however. Scrum requires that people estimate how long they think a task will take and then compares their estimate to the true amount of time taken. In theory people are not supposed to be evaluated based on their accuracy of prediction, but anyone in management knows that that doesn't happen. You can't help it. You will judge people's ability based on how accurate they are at predicting. People that are more accurate are going to seem better and people who are less accurate seem worse, because accuracy is the measurement being used. This results in the conclusion that the most brilliant people are often the people who are the worst predictors. It's often attributed to a kind of "mad genius" theory where the super smart person doesn't have a good practical mind.

Here's why that is stupid.

Of course the best people will be the worst estimators. They are working on the hardest, most complex things, which are, by definition the hardest to predict. In all likelihood they are better estimators than your "worst" people. If you want to find out, simply switch the tasks up and watch how accurate the best developer becomes and how hopelessly the less great developer twists and turns to even figure out a reasonable estimate.

Halos are dangerous and we manufacture them all the time. In reality, there is much more random in-determinism involved in project development. Very smart people can make decisions that lead to disasterous consequences. They can do so in good conscience and they can do it based on rational evaluation of the available data. Thus we should not be so quick in damning (or praising) people.

No matter how hard Obama tries to tell people that the economy won't be fixed overnight, or how he can't fix everything on his own, or how the problems are complex and people need to be patient, it won't stop the falling halo. He will just have to face the drop and hope that the timing of it doesn't hit near the next election... as for us, we should make sure that we are doing "better" than we were before when that next review comes around.

Interesting Video

here: http://www.edge.org/3rd_culture/kahneman_taleb_DLD09/kahneman_taleb_DLD09_index.html

Basically it's a nice explanation of some fundamental reasons for financial crises. Daniel Kahneman conducts some of the most interesting social/psychological experiments I've ever seen. While I may seem far too obsessed with social psychology for someone who makes games, I think these points are the same issues that infiltrate most human endeavors. Either way, I feel that their points are unusual since the market collapse is normally evaluated along economic or structural lines as opposed to psychological lines. This is a core flaw with analysis of most human social structure. We tend to say things like "the market wants this" or "management wants that" or "the audience is looking for such and such." Doing so ignores that these group abstractions are made up of individuals; individuals who do not follow rational or logical patterns.

Compression, Meaning, and Data

Information is bad for us. We are not good at remembering it, using it, or attributing it appropriately. This seems counterintuitive and I must admit that I am engaging in hyperbole; but I do that for an express purpose and that is the difference between meaning and data.

Data in its raw form is almost impossible for us to remember. Imagine a book with randomly written words distributed across 400 or so pages. Very few people would be able to remember much of anything. Conversely imagine the same 5 words repeated over and over again for 400 pages. Almost anyone would remember the data in the entire book. What is this so? It's because in the second case we are able to compress the data from 400 pages into 5 words. Even if we randomized the 5 words we could still compress all the data very quickly and remember the entire contents of the book.

Ok. This is a bit obvious and doesn't seem to have significance, however if we look at the implications of this situation we can quickly see what data can be so dangerous.

What we remember is stories. Stories compress data into easy to remember chunks which provide us with meaning. This creates interesting problems. For example consider the statements, which one is more likely to be true?

1) The king died, then the queen died.
2) The kind died, then the queen died of grief.

Most people select 2. This is because it has a causation attached to it and is thus more meaningful to us. Obviously, the answer is 1, since 1 contains the possibility of 2. This is what I mean by data is dangerous. We create causes all the time. We look for reasons and meaning whenever we are faced with data. The problem is that we are not particularly good at evaluating whether the reasons are true or false. Just because something makes sense, doesn't mean that it is what occurred.

Consider the above sentences again. Knowing that the kind and queen died is the available data. When we know WHY the queen died, we are much more satisfied, and the statement "makes sense." Of course the queen may have also died of disease, poisoning, falling down the stairs, old age or any other cause. The point is that we want to know the cause, because of meaning problem.

This makes us very weak to understanding things as events that do not have causes. For example, any small move in the stock market will result in a story explaining why it has moved. When Saddam Hussein was captured there were stories by the same news dispatches explaining why the market was up on Saddam's capture at one point in the day, then explaining why the market was down at another part of the day. Thus the same event was used as an explanation for competing results - obviously this must be false. The point is we can come up with reasons for just about any set of data. We can tie together data in order to tell a compelling and reasonable story, but that does not mean that we have identified what has actually happened. It takes much more effort NOT to narrate data into a story than it does to weave a tale that makes sense of things.

Next time you are in a meeting or read an article or are sitting at a family dinner watch how often people place causality on different pieces of data with no respect for evidence. You will find that if you try and convince people that it's not possible to know why such and such an event happened or that it was random you will encounter a lot of resistance. On the other hand if you simply offer a better story, then people will tend to believe you without too much questioning.

What use, postmortem...

Spilling holy grails is a dangerous business so a game developer questioning the value of postmortems seems a bit silly. Especially considering I spent a lot of time doing one - a good write up is here (http://www.tentonhammer.com/node/7919) if you're interested.

Anyone who has done a project and then done a postmortem knows the drill. You reflect long and hard about what you did come up with a list of "what went right," "what went wrong," give some thanks to people who helped you out and finalize it with saying what a great educational experience it was. Now all of those things are true and can be highly entertaining and at least somewhat useful... at least so it seems.

The problem I see is that if postmortems are so great and useful, why are they constantly filled with very similar what-went-wrongs? I have a theory that you can tell how practically useful advice is by looking at the volume of content on the subject. For example I assume that most financial advice, dieting plans, and self-help psychological books are worthless. I deduce this by going to the book store and seeing tons of books on those subjects. If any of them worked it seems like the solution would present itself, we would all be thin, well-adjusted financial wizards and move on with our lives. Since that's not the case the market is nice and hungry and the supply endless.

This is exactly how postmortems work. They are everywhere, and in all industries that have any kind of risk associated with their product development. In fact, the greater the risk of the project the larger the glut of postmortems. This makes sense. The more risky an area is, the more things go wrong, the more stuff there is to write about so that experts can feel smart and important. If you're wondering who the expert is, it's easy - it's the person standing behind the podium opposite the audience at conferences :).

Ok. Back to postmortems. Here's the problem. Analysis happens backwards with perfect information, ddevelopment happens forwards with limited information. It is very easy to put together the events that lead to a result after the events and results have taken place. It is almost impossible to predict the results when you are making the decisions based on limited knowledge and time. Thus, the lessons we learn in hindsight often do not apply in real-time because making the right decision often requires the knowledge of the result.

Here's an easy example. Imagine you are in charge of air traffic security in June of 2001. You are worried about how lax things have gotten. You rally hard for massive increases in airline security. You fight your superiors and stand before senate subcommittees to ensure you get your funding. You get lucky and win your support. As a result... nothing happen. September passes without a major iincident. A few months later you are accused of wasting funds on useless security programs, you are told that you have fattened up your department for your own gains, you are labeled power hungry and incompetent.

In the opposite scenario you don't rally hard for support and the events unfold as they actually did. Now instead of being labeled as a waster of federal funds you are accused of being lax. You are told that all the signs were there but your lack of imagination lead to a preventable incident.

It's a lose/lose scenario precisely because we don't measure the averted disaster. That extreme example illustrates the problem with postmortems. How can we actually know if the mistakes we made were not the best of a bad situation vs. something that could have been avoided? How can we know that our successes and failures are based on carefully crafted decisions with predictable outcomes, and not just luck? I'm not saying that good games are the result of luck. They aren't. What I'm saying is that they are the result of thousands of little decisions made by all the people involved day in and day out and to water that process down to a few bullet points may not be an adequate analysis. Because we are dealing with highly complex systems with many human inputs each decision has an unknown impact and consequences. You have a kind of general ability to steer projects in one direction or another, but the final results remains quite mysterious.

What we CAN learn for postmortems is how people respond to high risk and unknown situations. I think it is very important to evaluate what we do and why we do it so that we can try and identify those situations when they come up again and so, if we work really hard at it, at least try something different next time. That being said, the fact that postmortems bring up similar what-went-wrongs over and over again is evidence that learning those lessons isquite difficult. This is because we are confirmation seeking organisms. Once we think we "know" something we look for reasons to prove we are right... we don't like the nasty task of proving ourselves wrong - even in the face of overwhelming evidence. I think it would be really good if "soft" activities underwent falsification tests - that we were skeptical of what it is we do and do not understand about game development.

My favorite postmortem type things are done by Min Kim at Nexon. He talks on lots of panels and does interviews and is really open with stuff like *shock* financial details. He's also really humble with his answers often saying bizzaro things like "I don't know" and "9 times out of 10 the ideas you’ll have at the beginning will be wrong." Consider this articles: http://www.gamasutra.com/php-bin/news_index.php?story=17287. Notice how Kim is not arrogant about his answers at all. Can retail come back in Korea? Maybe... Maybe not. "I don't think games necessarily have to be made at [high] cost... I'm guessing 5 years down the line, it's going to be a multi-billion dollar industry in the US..."

Look at how introspective, open to falsification and non-committal that answer is. I don't work for Nexon. I don't know anything about their internal operations, and aside from brief hello-goodbyes and conference talks I don't know Min Kim. I believe, however, that with that kind of attitude Nexon is likely to identify new market trends because they willingly admit they don't know and they can't be sure. Instead, they measure and react. They try this thing and that thing, looking for what it is that works and are not caught up in being right and knowing for sure. In the context of the postmortem it would be like saying, some things went right, some things went wrong - we think this was a good decision, we believe that was a bad decision, we're not really sure. In the end the product was successful (or not successful) and hopefully we can be open to learn from our mistakes and not repeat them.

I have come to believe that the best method involves what Nassim Taleb calls "stochastic tinkering." Anyone who has been in the development pit has seen the result of that weird random thing some guy or girl tried and it was just really cool. In most cases those really cool things are not part of a plan or a design doc or any such thing. They are just kind of random results that can't be predicted or generated. They are pure bolts of lightning. When those things happen you have to be there to identify and incubate them. I call this chasing the cool and it's the best thing that happens during development.

In closing I will state that I love postmortems, I read almost all of them I can find, and I suspect at some point I will do more of them myself. I will also remain skeptical of their practical application and value, which seems like a contradiction - but how can we ever learn if we are unwilling to question our own assumptions?

Why our projects are always late - 20% vs. 500%

A topic of huge debate in the gaming industry is why games always slip?

If you go to any gaming news site and type the word "slip" you will find many example of high profile games that slip for many months, and occasionally years. Why is this? Indeed, a thesis could be written on this subject and considering how common this problem is, I was surprised at how difficult it was to find any specific research on the topic.

Now don't get me wrong, there are TONS of organizations dedicated to improving this problem: The PMI, the Center for Business Practices, the Six Sigma people and on and on. Other than the Standish Group's Chaos Report (http://net.educause.edu/ir/library/pdf/NCP08083B.pdf) there are few often cited studies into this particular problem (I have my own suspicions of that report because they do not post their raw data. I understand why they don't but social science teaches us that without the raw data it's not easy to evaluate conclusions - so I put that report aside).

So to cut to the chase, I encourage you to read this: http://simula.no/research/engineering/publications/SE.4.Joergensen.2004.b/simula_pdf_file.

This is the only paper I could find that specifically addresses at least a part of why games are late. In case you don't want to read that long paper, it works like this. There are 2 groups of software developers - professionals and students. They are split into 3 roughly equal groups, a control group a "low expectation" group and a "high expectation" group. They are each given a software development task to estimate. The control group is simply given the task without any input. The "Low" group is told that the customer believes that the project is about 50 hours of work but has very little knowledge of the project so the customers expectations should be ignored. The "High" group is told that the the customer believes there is 1000 hours of work, but that they have very little knowledge of the project so that should be ignored.

That's right. Same project, one customer expecting 50 hours, one expecting 1000 hours. The developers are then asked to perform the estimate as well as estimate the impact of the customer's expectation.

Now some results. Both professionals and students took about 30-40 minutes to estimate. The students averaged 10 activities the project required, while the professionals estimated 14. Roughly half of the developers responded that their estimates had been slightly impacted by customer expectation, but none of them indicated it had a large impact. The typical estimated impact was 20%, which means that if the developer indicated 100 hours, the customer expectation might deviate that estimate to be 80-120 hours instead.

The estimates were not quite so similar.
Student Group Result
The control group estimated median effort of 224 hours of work. The Low group's median was 77 hours and the High group's median was 404 hours. That's right. By simply making a small comment and also TELLING the developer to ignore it the difference between low and high estimates is five times - a bit more than 20% I'd say.

Professional Group Result
For the professional group results were similar. The Low group median effort was 77 hours and the high was 632 hours! Eight times as high. Median effort for the control group was 176 hours.

Oh, also, the median actual work required was about 250 hours so not only was the control group (i.e. non influenced group) much more accurate but students and professionals are about equal in their ability to predict difficulty in this case. So it doesn't seem that experience isolates you from the customer expectation impact.

The reason games slip is because developers don't estimate accurately. That is obvious. A big reason they don't estimate accurately is because they are influenced by all kinds of things. I can tell you from personal experience that I've never been in a meeting where the customer (i.e. publisher - or the one paying for the project) says "look I really want this game next year, but I don't have any idea how long it really takes, so just let me know what you really think." I'm sure you can guess that the expectations are more firm. Generally it's more like this "We need this game in 12 months, can you do that?" Of course we can. Then begins the dangerous thinking of things like, oh we'll just add 20-40% in order to "be sure" when in fact we might be off by 500%. Oops.

I'm not blaming publishers for pressuring developers and that being the sole cause of late games. There are lots of causes. This one, however, can be avoided if people who make decisions and create conditions for estimates were aware of how subtle comments regarding expectations can effect the estimate that comes back. If you want an honest estimate you cannot seed the estimator with expectations of any kind. In many cases you don't have to say anything. If the programmer or artist knows that the company needs work badly to survive, that will have a similar impact. If the estimator really wants the project because it seems really cool, that will impact the estimate. Protecting estimates against such outside forces is a difficult and deliberate effort and considering that most people are not aware of the impact it's no surprise that complex software projects, such as games, are likely to continue to be many months and years late - even after hiring more people AND going through brutal crunch periods.

That 500% error is really expensive!