Project Update: Wiki-Golf

Hi all,

This project update will be a little bit longer than normal since there are quite a few things to report.

First and foremost, I have created the git repo for wiki golf and hosted it on GitHub here. Feel free to fork, edit, and of course submit pull-requests. Although the code does not reflect this yet, I have chosen to distribute the package under the MIT License, so do with it whatever you choose.

Upon examining the repo, you will probably notice something glaringly different: The source is in Python instead of Javascript. This brings me to the next major point in this post, I am not using NodeJS to build the back-end of this project. As I started to implement the server, I was continually running into issues with the asynchronous nature of Node, and trying to force synchronous behavior in order to implement Breadth-First Search on the page graph.

I switched to Python, and encountered yet another issue. My first web framework of choice was the ever-popular Django. It took me all but a couple hours of skimming tutorials and documents to realize that Django was way too heavy-handed for my taste. It has way too much functionality that I would not be utilizing, and thus I chose to go searching elsewhere. It seems to me that the general consensus on building web-apps outside of Django with Python is Flask. Flask is a web micro-framework (micro being the key word here) that seems to be the ideal server implementation in this case. Using Flask allows me to only add the libraries and modules that I need, thus minimizing size and complexity, and increasing readability.

A couple of notes:

  • The setup-dev.sh script is not tested yet, so please give me a couple days to get it functional and tested. Or better yet, fork the repo and fix it for me (yay open source).
  • I do not plan on touching the front-end for quite a while. This is why there are several essential Javascript libraries that are not present (namely JQuery).
  • I do eventually plan on setting up the project so it can be handled by the google closure compiler, so please keep this in mind if you plan on making edits, as I will be expecting Javascript docs to follow this format as much as humanly possible.

Idea: We Pray

This post is a little out of the ordinary since it is not my idea, I’m simply posting so progress can be seen for the project.

At the beginning of this semester, I was approached by Bilal Bajwa and some other OSU students about an app idea that would stand to benefit the muslim student community at Ohio State.

The Problem: Those who practice Islam are required to pray five times per day as outlined in this CNN Article. This is not an issue within itself, however it can be when someone finds themselves in an unfamiliar place.  Though, even in a frequented public space, it can be hard to organize, and often muslims find themselves praying alone in the presence of others not familiar with the practice.

The Solution: Bilal has proposed a mobile application that helps students organize into groups for the 5 prayers. They envision a system where a students with the app installed will receive a notification before the call to prayer. After logging in and checking in, the student will be presented with a map and list showing the locations of other students and groups that have already started to form in the area. The student will be able to “RSVP” to any of the groups presented, or will have the option to create their own. In this way, we hope to help Muslim students organize for prayer in a much more community-like setting.

This post is going to be a little non-traditional in that some work has already been done towards the final product. I will provide some detail into what has been accomplished already, and where we would like to take the application as we move forward.

What Has Already Been Done:
A couple of friends and myself took this up to the “Kent Hack Enough” Hackathon last month in order to test out the possibilities of the system. We discovered quite a few things about our capabilities. We designed the back end in NodeJS using MySQL for data storage, and the front end was very obviously Android (we are saving iPhone for later). A few issues became very apparent very quickly.

  • Network calls in Android are an incredible pain. This isn’t the first time I’ve had to do this, however, I’ve never had to make so many different network calls in one application. All calls must be performed on a separate thread, and thus open the door to several potential race conditions, lots and lots of spaghetti code, and high indent levels.
  • Authentication is going to be a real challenge. Due to the planned openness of the system, security will be of the highest concern. Storing user locations in a database comes with its inherent security risks and one of the tasks that was put on the backburner due to time constraints was this.
  • Although the maps API in Android is much easier to deal with than I previously thought, half of the time was just spent getting the map to display. There are a bunch of issues that come up with the use of the compatibility library when displaying map fragments, and most of our time was spent trying to fix the same stack trace over and over again.
  • The Dalvik runtime doesn’t do nearly as much garbage collection as I thought. Due to the large amount of data exchange and storage being performed, we were plagued by memory leaks that couldn’t be traced to a source. We will have to be much more diligent about this the next time around.
  • Finally, I was not responsible for the back end implementation on this iteration. At the time of the Hackathon, I had never used NodeJS to design anything remotely similar to a WebAPI. There is a lot of code in there that I still do not understand, and MySQL is something I am very unfamiliar with. The back end is going to be implemented in a very different fashion this time around.

With all of the troubles, several good things did come out of the proof-of-concept.

  • We got the map to work without any issues. As I said, the API is very simple in general, so most of the map manipulations were one line of code each.
  • The back end did end up working. Although this part of the project will be overhauled in the next iteration, there is a lot that can be learned from the way it was done.
  • Although not functioning in the final implementation, many of the map manipulations were working very well in the end. This won’t be terribly much of an obstacle this time around.

How It Will Be Done:
From all of the lessons learned at the hackathon, I have a much better idea of how I will be implementing We Pray this time around.
In general, there is not much that can be done about the Android network calls issues (unless I want to make my own library for it). I will simply have to exercise a lot more discipline with the extra time that I’m given.
In general, it won’t be terribly hard to just do the back end in NodeJS once again. Now that I have a little more experience with it, I can have the advantage of designing it around the needs of the mobile client as opposed to the other way around.
The data in this case is going to be highly relational. As much as it makes me cringe, the DB would best be done with some type of standard SQL implementation. All we really need is a user table, and a group table, and that’s honestly it. Some type of Cron-Job will need to be set up to clear out locations after certain events or time intervals.
The nature of the daily prayer is that they change every day based on the times of sunrise and sunset. I foresee the need of potentially two services here. The first service will be used to determine the prayer times for the day, so that they can be routed to the mobile devices as a notification. The second service I will need is a way to determine prayer times based on location. It is very possible that both of these exist in the same api, but this will definitely require some searching.
The final issue will be iPhone. I have never done iPhone, so I will either need to phone a friend for this, or swallow my pride and learn it. There really isn’t much else to be said about this.

This project’s source will not be made public until it is fully functional and operational. As per usual, I will be hosting the git repo on BitBucket, however this one will be private. I will also be diligent about posting updates since I will be the only one with access to the source.

Cheers,
Dan

Idea: Wikipedia Golf

So recently, I heard my friends speaking about wikipedia races.  This is essentially a game where two players select two random articles on wikipedia, and race to see who can reach the second article the fastest using only the internal links.  This sounds all fine and dandy, but it seems a little unfair to whoever happens to be the slower reader.

Here is what I envision:
I picture a golf-style game where users are on a “course” in which each hole is a challenge to get between two pages on wikipedia using the internal links. I aim to make the races more of a mental game than a speed one. At the beginning of the game, the user is presented with the challenge, and given a par score (calculation explained later). There will then be an embedded webview displaying the content of the first page. The user can click any link on the page, and each click is registered as a stroke (I will have to find a way to prevent cheating). There will be either 9 or 18 holes, which I will decide later once I figure out how long a hole generally takes.

Implementation:
I’ve been looking at NodeJS for a while now, and I can’t wait to get my hands dirty. I plan on using node as a back end with it’s extensive modules that will provide for graph search capabilities. I’ve been looking into using a database for storing the data (maybe not the most effective way, but I feel it is worth the practice) and since this will mostly be taking advantage of the graph-like structure of wikipedia’s layout, I will be using neo4j. Neo4j is a graph database implemented in Java on top of a NoSQL-style db. I will be using neo4j’s Cypher Query Language to pull data and run searches for calculating par.

The Graph:
The wonderful thing about the wikipedia layout is that it is practically already a graph on its own. Each page serves as a “vertex”, and each link serves as a directed “edge” between vertices. This unlocks a whole bunch of useful properties through simple graph algorithms.

On The Topic of Calculating Par:
Going off of the previous section about exploiting specific graph properties, par can be calculated rather simply. This is the classic problem of finding the shortest path between to points on a graph. I will be using breadth-first search to find and store this path. To make under-par scores possible, I will be adding 3-5 “strokes” to par in the interest of fairness.

Data Collection:
The WikiMedia API provides a great set of tools for gathering the page content of articles in wikipedia’s special markup language. The markup language shows the internal links in the form of the string [[TITLE|PAGE_NAME]]. These can be parsed out using simple regex matching, and the titles can be fed back into the algorithm recursively to find the next page.

Potential Challenges/Issues:

  • Recursion can be very dangerous when dealing with a dataset of this size. This will be abusing the stack HEAVILY with (potentially) very deep recursion, so I may have to find ways to do this without (I cringe at the thought though). On top of this, recursion can be tough to pull off inside an asynchronous programming model. Which brings me to my next point.
  • It is potentially very easy to find myself in Callback Hell. First of all, NodeJS has a way to get like this on its own with the asynchronous execution model. A lot of the operations I would like to attempt run synchronously, so waiting for threads to finish may become a realized nightmare. Secondly, code readability can be nothing short of a disaster when the tab level gets this deep. I will need to practice extreme discipline to keep this from getting out of hand.
  • In general, this will be very hard to test. Without a solid way to visualize the data, I can easily tell that anything calculated is coming out correctly. This will be something I plan on addressing in update posts.

I will be posting the link to the git repo on BitBucket as soon as I have something workable up. Please feel free to contribute via pull requests and the like. In general, the majority of my projects will be public and open-source.

Introduction

Hello All,

Looks like OSU has given me a good opportunity to start doing some project posts.  I will be using this page mostly for descriptions and updates on the various projects I’ve started on in my adventures as a Computer Science + Engineering student.  I’m trying to accomplish two things here.  First and foremost, this will hopefully by an employer-facing view of my current, past, and future personal/school projects.  Second, this will hopefully help me organize my own thoughts, as I will be posting ideas that I have before I implement them.

Posts will generally come in the following forms:

Idea Post:  These will generally be pretty free-form.  They will start with a description of the use case and benefits.  I will also try to explain the source of the idea so we can all get a hint of my inspiration.  The rest will be the free form, I will most likely discuss any research I’ve done, share links to similar pieces of software, and possibly go into some detail of how I plan to implement the whole thing.

Project Update:  I will generally try to have 2 or 3 of these between project introduction and completion.  I will do one of these if I stumble across anything important, or if there is some kind of epiphany in the process.

Project Post:  These will be a bit more structured as they represent a complete project.  I will reference the original idea post if it exists first, this will hopefully show some continuity between projects and also give a decent idea of how long the project took from development to completion.  There will also be links to demos, resources, and any tutorials that I may have found useful.  Finally, there will most definitely be a link to the finished project, and a link to it’s associated Git repository.

Cool Technology: If I stumble across anything exciting, I will definitely be sharing.

I will definitely be updating this post as more things come about, and I’m just starting, so this is clearly tentative.

 

Cheers,
Dan Marchese