Abstract

Before any software product can be released to the general public, it must be thoroughly tested to ensure that it functions as originally intended.  For a piece of software that will be used in medicine, for example, testing a product thoroughly can mean the difference between a patient's life and death.  Testing is just as important in the gaming industry.  Although it might not be a matter of life and death, the shipping of a faulty product can hurt a company's sales and even tarnish the company's image.  Delivering a game that is both playable and bug-free is the goal of every game company in existence today.  With the complexity of games and with the standards consumers have come to expect, the testing phase in a game's development process has become one of the most integral, man-hour-intensive jobs in the software industry.  This paper will provide an in-depth view of all that goes into the testing phase during the completion of a video game.  The various levels of testing will be examined, as well as what can go wrong when lapses occur during these phases.

 

Introduction

In today's computer-oriented world, the word "bug" is used to refer to an error or fault that occurs during the execution of a software program.  Game developers, as well as developers of any piece of software, strive to produce a product that is as bug-free as possible.  Contrary to popular belief, the derivation of the word "bug" can be traced all the way back through history to Thomas Edison's era.  Even as early as 1896, an electrical handbook from Theo. Audel & Co. used the term when referring to a fault or failure in the workings of an electrical apparatus [3].

 

However, the history behind bugs and their relation to software and computers became most popular on Sept. 9th, 1945.  During the testing of Harvard University's Mark II Aiken Relay Calculator, a moth was found trapped in between points at Relay #70, Panel F [7].  The operators of the machine placed the moth in their computer logs with the entry "first actual case of bug being found."  They then went on to state that they had successfully "debugged" the machine, thereby coining the phrase "debugging a program" that is used in today's software industry.

 

Although the bugs found in today's software programs are no longer of the six-legged variety, these software defects still pose a large problem for developers.  A video game with terrific design, story line, and graphics will still not sell well if it contains major bugs.  Thus, a large amount of time and effort is devoted to the testing phase of a video game's development process in order to create a game that provides the user with an enjoyable playing experience.

 

 

 

Previous Research

The art of testing software thoroughly is a well-documented one due to its need in everyday software development.  "Software bugs will almost always exist in any software module of moderate size: not because programmers are careless or irresponsible, but because the complexity of software is generally intractable - and humans have only limited ability to manage complexity" [5].  All companies have their own series of functional, system, and performance tests that a product must undergo.  Video game companies are no different in that they put their games through a period of vigorous testing.  If a game hits the market with serious flaws and defects, magazine reviews will note these problems and the game's sales will suffer as a result.  There have been instances in the past where a game company failed to test its product thoroughly, leading to game recalls and heavy financial losses [1].

 

The Game Development Process

From a very general overview, the development cycle for a video game, from its conception to its completion, goes through four major steps: the pre-production phase, the production phase, the testing period, and, finally, the point where the game is manufactured in mass quantities and distributed [2].  Depending on the game, this process can take anywhere from 1-2 years to complete, sometimes even longer.  The "Madden" series of football games developed by Electronic Arts, for example, must be ready to be sold at the onset of every football season, and, so, they cannot take more than a year to develop.  Other companies have different time restraints.  For example, Valve Software's much anticipated "Half-Life 2" game is set to be released this Fall; it has been about six years since the original "Half-Life" was released [4].

 

The pre-production phase in a game's creation is also known as the development or prototype period.  Usually lasting between 3-6 months in length, producers and designers work together during this brainstorming period to create design documents and technical specifications for possible game ideas.  Possible members of the teams working on the game are selected, and estimates are made as to what the game's budget and schedule will look like.  If a new game engine is being developed, or if other new tools or technologies will be used by the video game, this development phase can often last up to one year or even longer in length [2].

 

It is during the actual production phase of a game's development cycle when the features that were originally described in the design documents are turned into reality.  This time period is a constantly evolving process in which features (such as level design, character development, etc.) are added, cut, or modified depending on memory, time, and technology restraints [2].  Taking on average about 1-2 years to complete, the production phase usually has 3 major milestones that it tries to reach: the prototype, alpha, and beta versions of the game.  The prototype is meant to merely provide a sense of the overall gaming experience that the video game aims to provide.  The alpha is the first actual playable version of the game, even though all of the game's graphics and data might not be available yet.  A game is considered to be a beta when all of its features and assets have been completed, although there may still be bugs and game play balance issues.

 

The testing phase during a game's creation is a complex and challenging process that usually lasts about 3-6 months and sometimes even longer.  Every possible path a user can experience must be tested by the team, and, if the game can be played online, all multiplayer aspects of the game must be tested as well.  The actual testing stages will be described in length below.

 

When a final copy of the game is ready for release, and the code has been checked for viruses, a "gold CD" is created and shipped to the manufacturer (for console games, the manufacturer must be approved ahead of time) [2].  The first few CDs that are produced by the manufacturer are tested against the original gold CD to make sure that they match before production is continued.  During this period that usually lasts about 1-2 weeks, the retail packaging and final documentation is created until everything has been assembled at the warehouse, and the game is sent through distribution channels to the various stores [2].

 

The Testing Stages: From Alpha to QA

As stated above, the testing phase is a very complex and challenging time period for game development.  If a video game isn't thoroughly tested, and it is released with defects, sales will suffer, and the game company itself (depending on how well-established it is in the gaming market) may lose part of its fan base.  Although informal testing occurs throughout the game's creation by various programmers and producers, the period of formal testing is divided into four major stages: alpha testing, beta testing, configuration testing, and quality assurance.

 

The alpha testing stage is the real "make-or-break point for a game" [1].  Just as the name implies, testing is performed on the game's released alpha version.  Usually the alpha includes only some of the game's features and only parts of the game will be functional - a large portion of the video game's graphics and data is still yet to be included.  At this point, the game's producers and developers thoroughly examine the alpha in the attempt to determine if the game will be "fun enough" to succeed in the marketplace; mediocre products may get killed in order to save money that would otherwise be spent on marketing and continuing to create the game [1].  If the game passes its inspection, development moves forward and the real alpha testing begins - a period that can last several weeks or even several months.

 

During alpha testing, a test manager develops an overall test plan consisting of the various test cases to be run.  The game's producers and developers then hire testers or assign people from other projects to execute the created test plan.  In total, there may be between 25-30 testers working on the alpha version of the game at one time [1].  The testing manager will set up a bug database in which found bugs will be logged by the testers.  The programmers will then deal with the bugs listed in the database and try to fix them.  The reported bugs will remain "open" or flagged after being fixed by the programmers until the problem has been retested and verified as being fixed by the testers. 

 

Once all of the game's levels are designed, all of the game's content has been included, and the game's art and audio has been created and is in place, the beta testing can begin.  Every aspect of the game must be tested at this point - each feature on every level must be examined using every weapon, game object, airplane, or whatever else may be included in the game.  Since such a task can be daunting and very time consuming if it is done in-house only, publishers can choose to allow open beta testing on the project.

 

Open beta testing refers to the usage of members of the general public to perform field tests on the game.  This sort of testing is only possible for PC games, however, as customers outside of the company lack the required hardware that is necessary to operate a beta version of a console game.  In most cases, publishers provide external testers with a beta version of the game that will work for a limited duration of time only, as well as requiring the testers to sign a contract not to make copies of the game; this is meant to help prevent piracy of the game [1].

There are several benefits that a publisher can attain by allowing open beta testing.  For one, outside beta testers may attempt things or try out things in the game that weren't thought of previously by the programmers or in-house testers.  Furthermore, it provides a good opportunity to evaluate how the game performs with different hardware configurations from what the internal testers had been using up to this stage in the process.  Lastly, open beta testing can be a form of marketing in itself.  If the beta performs well and meets the expectations of the various users, this can raise the enthusiasm for the game's release [1].

 

Some of the problems that can occur by using open beta testing are a direct result of using inexperienced external users.  Since the external beta testers are not being paid by the game company for their services, they will most likely not be as thorough as testers who are actually working for the company might be.  Secondly, bugs that are reported by these testers may not be documented or described well enough for the programmer to fix them properly [1].  Also, some of the feedback from the external testers might not actually be that of bugs that were found.  Some of the so-called problems might just be design features or aspects of the game that the users did not like.

 

After the beta testing has been completed, efforts can then be focused on the next stage of the testing process, configuration testing - a phase that is absolutely necessary for games created for the PC, less for those created for consoles.  The differences between consoles and PCs are many.  Developing a game for a console is a lot easier than developing a game meant for a PC.  For consoles, every machine that is produced is shipped with the same standardized controllers (unlike a PC, where a user can choose from any number of differing game controllers and joysticks) [6].  Developing a game for a console is also more attractive to a company due to how inexpensive it is to purchase a console compared to a PC.  The lower price practically guarantees that there will be a larger number of users who own the console as opposed to owning a PC [6].  This also means, then, that there will be a larger market for console games.

 

The main problem facing any company developing a PC game is the simple fact that no two machines are alike - there are millions of different possible hardware configurations that PCs can have [6].  Games on a PC also tend to "push" the machine a lot harder than any other application.  This can lead to configuration conflicts.  Other aspects of PCs that developers must be aware of are the fact that a mouse has a lot more precision than a game controller, a computer screen has a much higher resolution than a TV, and that a PC is much more likely to have an Internet connection than a console [6].  All of this leads stresses the importance of configuration testing of PC games.

 

Configuration testing tends to begin towards the end of the game's production cycle.  At this point, the game is more stable, and the developers can focus on testing different hardware setups and different operating systems [1].  Most game companies have a special configuration lab in which different video cards, audio cards, types of memory, and processor speeds can be studied to see what sort of an effect they would have on the game play [1].  This sort of configuration testing also helps to determine the minimum hardware requirements that will be needed to effectively play the game.

 

Quality assurance (or QA for short), is not only the last stage in the testing process that a game must undergo, but it is also the most hands-on of all the different testing phases [2].  The game goes to the QA team when all of the recorded bugs that were found have been fixed, and the producer thinks the game is ready to ship.  However, "if a game ever crashes, responds inappropriately to a command, or displays something it isn't supposed to display, it fails QA and cannot be shipped" [1].

 

The team in charge of quality assurance for a video game is oftentimes separate and independent from the production team [1].  Therefore, the QA testers will be both unbiased and not pressured into clearing a game for production that still has problems.  Besides checking for crashes, command failures, and the display of incorrect information as stated above, the QA team is in charge of making sure all the details in the manual are correct as well [1].  Images on the screen must match those used in the manual, and the testers must make sure that the game's commands work as described and are documented correctly.

 

Whoops: When Slipups Occur

Despite the amount of testing that goes into a video game's creation, game companies have not been without their fair share of bungles.  In the summer of 1994, following the success of the movie title, Walt Disney began working on developing a computer game version of The Lion King with the intention of releasing it in time for Christmas [1].  At that same time, however, a hardware company specializing in CD-ROM drives and sound cards produced a new standard for a "Multimedia PC" [1].  Basically, it was a PC machine with an 8-bit sound card a single speed CD-ROM drive [1].  Since the Multimedia PC was essentially the newest thing in audio and video for PCs, sales skyrocketed.  Meanwhile, Disney had decided that the "Lion King" game would sound best if it were to be used with a 16-bit sound card [1].  Needless to say, the game did not work on the Multimedia PC.  As a result of this configuration testing failure, thousands of parents ended up calling Disney on Christmas morning to complain how the game would not work on their computers.  Over half of all the games that had been sold ended up being returned for refunds [1].

 

In January 1999, the game company Electronic Arts had to recall over 100,000 copies of the new "Tiger Woods 99" Playstation game [1].  Apparently, somebody had accidentally placed an AVI movie of South Park's "Spirit of Christmas" episode on the disc [1].  Although the movie could not be viewed on the disc by playing the game, the real embarrassment was that EA didn't even own the rights to South Park - another company called Acclaim did [1].  A simple QA check to verify the purpose and identity of every included file on the disc would have saved EA a lot of money.

 

Summary

Before any software product is released to consumers, it must be thoroughly tested in order to ensure that it functions as intended.  Video games are no different.  With the complexity of computer and console games today, as well as the standards gamers have come to expect, the testing phase of a game's development process has become one of intense effort and focus.  A game is put through endless hours of alpha testing, beta testing, configuration testing, and quality assurance testing before it can be released to gamers worldwide.  A video game will simply not sell if it contains significant bugs and flaws.  Therefore, a lot of emphasis is placed on creating a game that is both enjoyable and playable.  Without the presence of a thorough testing process, this would not be possible.

 

References

[1] Adams, Ernest.  Break into the Game Industry: How to get a Job Making Video Games.  San Francisco: McGraw-Hill/Osborne, 2003.

 

[2] Gershenfeld, Alan, Mark Loparco, and Cecilia Barajas.  Game Plan.  New York: St. Martin's Griffin, 2003.

 

[3] Huggins, James S.  First Computer Bug.  21 Sept.  2004

<http://www.jamesshuggins.com/h/tek1/first_computer_bug.htm>

 

[4] Newman, Heather. "Half-Life 2 Worth Long Wait.  19 Oct.  2004

<http://www.technewsworld.com/story/Half-Life-2-Worth-Long-Wait-37451.html>

 

[5] Pan, Jiantao. Software Testing.  20 Oct.  2004

<http://www.ece.cmu.edu/~koopman/des_s99/sw_testing/>

 

[6] Rollings, Andrew and Ernest Adams.  Andrew Rollings and Ernest Adams on Game Design.  Boston: New Riders, 2003.

 

[7] The First "Computer Bug".  20 Oct.  2004

<http://www.history.navy.mil/photosimg/h96000/h96566kc.htm>

 

Note: views and opinions expressed in this article are the author's and are not necessarily those of Wolverine Soft.