Training an AI to beat Pacman

Joshua Sandvick
STEP Signature Project
1/30/17

The main activities of my STEP Signature Project were to first build a powerful computer, and then use it to train an Artificial Neural Network (ANN) to beat the first three levels of Pacman. Building the computer entailed lots of research to pick the correct parts, followed by actually assembling the computer. Training the ANN required me to use my machine learning experience that I gained from my summer internship to find the best way to teach the network to play Pacman.
The aspect of the project that was most transformational to me was the fact that so many of my assumptions coming into the project were wrong. One example of this was when I thought that assembling the computer would be much easier than training the ANN. While training the ANN was not easy, I found assembling the computer to be at least as hard if not harder than training the ANN. Since I had budgeted my time mostly for the ANN training, it was quite a shock to have to pour over 15 hours in total into researching and assembling the computer. Since I’ve never had a chance to work on a project by myself that was well-funded before, I felt that I learned a lot about making my schedule for a project quite flexible even if I think that I have lots of knowledge about the project beforehand. Another transformational aspect of the project was focused just in the realm of training the ANN. I thought that it would take an enormous amount of time to write all of the code for the Pacman training, but as I will talk about further on in this paper, smartly using the open-source code available from other programmers greatly reduced my time needed on the project. I feel that even more than learning about the intricacies of machine learning, practicing how to find and use good open-source code is one of the absolute most important things that I can learn as a computer scientist.
I think that describing the whole assembling of the computer will best describe why it taught me so much about keeping my project schedule flexible. Initially, I believed that researching and assembling it would take about 6 hours at most. This assumption was shattered immediately at just the research portion, though. It turned out that there are a wealth of options for each and every computer part, and having very little knowledge about all of the different brands and specifications coming in, I had a very difficult time deciding on the correct parts that would best aid in training the ANN. But by using the internet and the extensive help of three very knowledgeable friends, I was able to learn about almost every computer brand for all of the different computer parts, and I made a very informed decision on every part in the computer. To be able to gain this needed knowledge took about 8 hours, which was way over the amount of time that I had budgeted for it. This led into the next part, which was assembling the computer once all of the parts arrived. I anticipated this step to take around 3-4 hours based on what I had read online, but the directions that came with all of the parts were extremely vague since computer designs are very modular, and it took me around 10 hours to fully assemble the computer and get it working. On the other hand, I had assumed that training the ANN would take between 10-20 hours and then days of training time, but it only took about 6 hours in total. These discrepancies in my allotted times really caused my idea of budgeting a schedule to transform. The computer can be seen in the picture below.


Again, I believe that explaining the training process of the ANN will help to describe what I truly learned from this project. After my computer was assembled, I began researching common ways that people have created artificial intelligence agents to play Pacman. Since I had a lot of experience reading academic papers from my internship experience, that was my natural realm to research in. I quickly ascertained that for numerous reasons, it would be much more feasible to create an ANN that could solve Ms. Pacman rather than Pacman, and thus I thought it prudent to do so. One particular paper described a very reasonable ANN training regimen, and they provided source code that contained a simulation of Ms. Pacman and a framework with which to setup the ANN training. I thought that was one of my best chances for getting good results, and my time that I had spent reading papers turned out to be well spent. Immediately, the source code allowed me to work strictly with the machine-learning aspect of training the ANN rather than deal with writing its framework, and I was able to get very good results very quickly. I had heard of the power of using open-source software before, but after having used it firsthand, I will definitely try to do plenty of research before every project and see if there is an open-source tool out there that will increase my efficiency. I felt that unlike the semi-uselessness of my time schedule that I had created for this project, the pre-planning and research that I did before even starting to try to train the ANN to beat Ms. Pacman paid off more than I could have imagined beforehand. The graph of one round of training Ms. Pacman can be seen in the figure below, with the x-axis representing the number of games played and the y-axis representing the percentage of pellets eaten in a game.


My first transformation, involving budgeting time for a project, will most likely be the more valuable one to me because it is applicable in many areas of life. I plan to run my own software company someday, and I will need to have budgeting time for projects down to an art if I plan on having that company be profitable. While I think that one of the big takeaways here is that sometimes your best estimate of time needed will be wildly wrong, I believe that if I had the right experts advising me before I started, I could have come up with a time estimate that was much closer to the truth, and so in the future I will always seek out those kinds of people and consult with them as much as possible. This lesson about time isn’t just applicable to a business, though, and I’m sure it will come in handy many times even with something as simple as budgeting a fun day with my future wife and children.
My second transformation, involving not trying to reinvent the wheel in terms of software tools, is also very important to me and especially to my career. Many computer scientists, very much including myself, have a certain pride about them that drives them not to want to use other’s software if it is not needed. With this project, I believe that I have a very strong case for myself to use open-source software when it’s applicable. It saved me such a large amount of time and got me such strong results that I would have been a fool not to use it. Now, I do believe that there is a time where you should implement something from the ground up in order to learn all of its intricacies, but this should only be done when necessary since there are often faster ways to learn things. I will take this lesson with me throughout the rest of my career, and it will make me a much better computer scientist.