More Website News

Leaderboards have been altered a bit; they now track percentages and also show the user's rank even if they're not in the top 10. 

The tutorial page has been changed a bit, although it could still be improved.

We added some predictions but we need more.  This is one of the most important things right now.

The ad has been completed and posted.  We don't have anywhere near the number of users we'd need to get proper data, though.  Maybe we should try advertising to Caltech?

HTTPS works as well as Automatic Admin Functions!

Thanks to enabling communication on port 443 (SSL) on our virtual server, HTTPS now works! Facebook users using secure browsing should now be able to view our app! That's one major obstacle solved!

We now have a new logo, which we'll use for our Facebook ad campaign. Our livesite has been updated with our changes.

Some administrator functions used to return bids when a prediction closes and endow money each day have become automated.

Now on to Week 5:

• More questions
• Tutorial
• Advertisement
• Storing/Collecting data for analysis
• Secondary prediction table: (meta info including links, link to an image/icon, last value)
• Perhaps a table to track prediction values over time
• Better Prediction interface
• Improvements to leader tables and user info

New server and https

We set up the new server thanks to Steven Low and George Lee; it works.  You can now play the live version of our game at apps.facebook.com/foresight_game. Unfortunately, we still do not have HTTPS working. The certificate appears to be setup well in Apache, but for some reason, the site won't load with https. Those using Facebook's secure browsing will just have to wait.

We also implemented the feature to remove bids. You get your reserved money back upon doing so. Upon the close of a prediction, users ought to get their money back for any unresolved bids; this has yet to be done.

There were also graphical and user interface changes on the test site that haven't been uploaded on the live site.

We're still working on the tutorial.

Buying, Selling, and the New Server

With php-back end and game logic, we have implemented the ability to buy and sell shares of predictions, along with the associated transferal of money.  You can also remove bids.  May need to do some debugging.  Very close to being able to decide predictions; all that's missing is the transfer of money.

Were given a new server thanks to George Lee and Steven Low; spent some time setting up http, ssh, php, and postgres.  Still trying to work out how to link it to the sites and stuff.  Will eventually be at 4sight.wil.caltech.edu .  Trying to figure out how to setup a database with the same name as our old one and no password required.

Bid Progress, Website Layout Changes, and Presentation!

Moving into the 3rd week, we have partially implemented bid resolution. One can post bids within their budget and they are immediately resolved with other standing bids. However, we do not actually complete any "monetary" transactions, so nobody gains or loses any shares of a prediction. There has been a holdup, but we believe we are very close to completing this. Because we have a list of currently unresolved bids, we can estimate the market price of a prediction based on the lowest Sell price and highest Buy price.

Thanks to lots of CSS styling, our site looks a lot better than before. You should check it out! http://ugcs.caltech.edu/~alexf/test.php

Finally, we are presenting our current progress and goals today to get feedback and other people's insights into our ambitious project.

Short-term Goals for Weeks 3-4:

1. Complete transactions so that users can actually own/shortsell shares of a prediction.
2. Administrator Functions to decide a prediction and award money.
3. Administrator Functions to grant money every day to all users.
4. Tutorial / Description of How to Play the Game.
5. More UI!
6. Security (and HTTPS?)
7. Create the real Facebook app with a sandbox on the side so that we can make changes w/o disrupting users.
8. Advertise our site on Facebook and get people to play it!
9. Think of fun questions to pose to the public.
10. Make some sort of friend invite system with the Facebook API
11. Allow people to submit links/comments about predictions. Alternatively, we can post information regarding them so that it is easier for our players to predict with more certainty.

More Website Stuff

Moving to the Home page from another tab is now noticeably faster than re-loading it; the website now only needs to load everything the first time you visit.  In addition, we added a new table to keep track of users' friendlists and only need to record those once.  There are a few more tables in the database for keeping track of some things, but they haven't been filled with stuff yet.

Also fixed a few minor errors.

Layout Changes and Future Goals

We decided that it would be better to split our single big page into a bunch of smaller pages in order to reduce future server load. Don't worry though, since our tabbed layout has stayed the same.

In the future, we will want to just have 4 main pages.

•  A Homepage listing some popular predictions
• The list of predictions and their current probabilities
• User's status (what prediction shares they own and what they are bidding on)
• Help (how to play, how to get more information)

When you access a single prediction, you will be given information about the prediction such as when the market closes, and what prices it is being traded for. You can then post a bid; if the bid can be fulfilled, it will be immediately transacted.

This last step needs to be done. We also want our site to look nice.

We also have issues with HTTPS still, which we are trying to resolve.

Where to put Leaderboards and how we're gathering data will be done later. Perhaps another database table could store the latter.

Since we're close to going live, we will want a playground/test area so that we can continue to make improvements without impacting our live site.

Viewing Predictions and Making Bids!

We now have an admittedly clunky interface to look at the current predictions and make bids on them via PHP/Postgres. We will eventually have more questions than the ones posted now. Take a look at http://ugcs.caltech.edu/~alexf/test.php.

More Website and Postgres Stuff

Website now automatically adds users to the users database.  Right now, for testing, it also prints out a list of all users and their stats.

Website also lists the predictions a user owns, but the database is still empty, so there's no way to test it yet.

Added a new page to add or remove predictions from the database as well as list all predictions.

Some cosmetic changes

We added some cosmetic changes to our blog, as you can see (template logo), and to our app - now we have a kinda tabbed interface. Considering another AJAX solution, look at http://ugcs.caltech.edu/~alexf/ajaxdemo. Still need to brush up some PHP/JavaScript/figure out this Facebook API to continue on with our grand quest of predicting the future and making this thing useable... Now working on a template for the prediction page.

As usual, you can check out the app at http://ugcs.caltech.edu/~alexf/test.php

Progress on Database and Website: Friendslist

Now able to retrieve friends lists of users at http://ugcs.caltech.edu/~alexf/test.php.
Used http://developers.facebook.com/docs/reference/api/ to learn how to extract the information.

Created tables with php and postgres according to schema in previous blog post.  Four tables named "users", "owned", "bids", and "predictions".  It turns out that the Postgres on UGCS is not fully up-to-date. We didn't check for sure, but it seems to be ~8.1, not 9. Also have a page with buttons to create and delete them for testing purposes.  Link not given.

Connecting with Facebook and Verifying Postgres

Using Alex's UGCS account, we were able to authenticate/authorize with the Facebook API, extracting the user ID. The Graph API is very useful for accessing this data. At the same time, we were able to access the Postgres database on UGCS by creating and dropping a table.

You can check out our Facebook test app at: http://ugcs.caltech.edu/~alexf/test.php

One interesting setback seems to be that UGCS does not support HTTPS since it lacks a trusted SSL certificate. Unfortunately, we will probably need to find a new location to host our site.

Week One Goals:

We intend to create the database and website; we want to have the basics of the website running by the next meeting on Tuesday.

The tentative schema for the database should be:

User Table

UserID, Authentication, MONEY, Reserve, # correct?

Give money every day. (First bid of the day)

Prediction-User Relation

UID, PredictionID, # that they own (can be +, -, or 0)[, numSell, bidSell, numBuy, bidBuy]

Prediction-User Selling Buying Table

UID, PredictionID, datetime, buyBool, #that you want, price

Prediction Table

PredictionID, Section, Subsection, Question, ClosingDate, DecidingDate, FinalResult

Compute CurrentOdds with a function (as well as max Buy and min Sell)

SELECT max(bidSell) AS minSell FROM pred-usertable WHERE PredictionID = predID

SELECT max(bidBuy) AS maxBuy FROM pred-usertable WHERE PredictionID = predID

 

The website should include

• A home page, which should have a list of all predictions
• A page for each prediction with its individual stats
• A page for bidding on the predictions
• A page for each user's individual stats

If time permits we can expand the webpage, but we plan to have at least this much up by next week.

 

For testing purposes we should fill the database tables with test data until we're ready to open them to the public.

Project Plan

Project Plan: Prediction Market
Group Members:
Alex Fandrianto
Daniil Kitchaev
Tuan Anh Le
Richard O’Dowd
Robert Shimizu

Introduction

In this project, we intend to create a prediction market game on Facebook and analyze its convergence properties with respect to time, users, information, and other factors. We know prediction markets provide interesting insight into group perception of the probability of an event. So, by analyzing how the market converges, we will be able to see the accuracy and speed of group predictions. An advantage of using Facebook for this purpose is that we will have access to a large, ready supply of possible users, and an existing application system providing ease of use and reach for our game. In addition, Facebook allows us to explore the effects of social network structure as it relates to information cascade, predictions, and attempts to manipulate the results.

For this project, we intend to partially follow the example set by Intrade, a currently existing successful online prediction market, in constructing our prediction market. Each prediction will be a yes or no question, and be offered in a continuous double auction format. Individuals will be given a set number of points per day which they can use in all standard market activities, such as buying, selling, short-selling, etc. Shares will clear for zero points if false, and some fixed number of points if true, so individuals will have points frozen in their accounts to ensure that all potential losses are covered.

The data we will collect for the market will consist of price data as a function of time, participants, total number of bids, what the answer is, etc. In addition, we will construct a network graph containing user information regarding their bid prices, bid times, questions answered, etc. with linkages reflecting friends on Facebook. This network component is important as it will allow us to track information cascades, clustering of similar opinions, and other information based effects.

The first step of our data collection will be gathering data from normal predictions without any modifications on our part. The goal of this is to form a baseline for our further analysis. This will also allow us to get user feedback, increase our user base, and introduce any modifications that we will need for later runs. We will then follow this up with experiments as time allows, such as an insider information and market bubble experiment.

The insider information experiment entails introducing a prediction that we control, such as coin flips or dice rolls. We then feed this information to certain users and see how this affects convergence of the market. From prior knowledge, we expect to see direct information cascades from these individuals to others in their friends networks, and indirect information cascades as unrelated individuals react to market changes caused by these “insiders”. Thus this experiment will give us insight into how information cascades propagate throughout prediction markets, and how social networks affect information in prediction markets.

The market bubble experiment will test the robustness of prediction markets when faced with attempted manipulation. To achieve this end, we will artificially modify a prediction by either buying or selling a lot of bids, modifying the current perceived probability. This experiment will thus allow us to test how quickly prediction markets recover from manipulation if they do at all. In addition, this will give us more information regarding indirect information cascades and how they react to false or manipulative signals in prediction markets.

For analysis, there will be two types of data being analyzed: aggregate and user. As a first step for aggregate analysis, we will compile metrics related to accuracy, convergence speed, popularity, user bid clustering, etc. We can then compare this information to the results of the prediction and other predictive mechanisms to see the effectiveness of a prediction market. User data, on the other hand, will primarily be analyzed from a social network and information based view. From the user data, we will be able to see how bid choices propagate over social networks, and how they change with publicly available information such as bid price and private information such as insider trading. These two types of data will thus allow us to make useful inferences about the convergence properties of prediction markets in social networks.

Literature Review

Cogwill et al. Using prediction markets to track information flows: evidence from Google. 2009. <http://bocowgill.com/GooglePredictionMarketPaper.pdf>

Google carried out an internal prediction market with its employees. Questions varied from topics relevant to the company and topics that were of interest to employees but irrelevant to Google. The data collected revealed several biases that tended to decrease as employees gained experience with the market. One bias was favorite bias, where players would tend to overestimate predictions likely to come true (favorites) and underestimate the odds of predictions unlikely to pass (longshots). We hope to replicate these results. Players were also generally optimistic in predictions that favored Google, but this bias is unlikely to appear in our public prediction market. Another difference is our prediction market model. Google did not allow short selling of stock; instead allowing users to trade both positive and negative securities of a prediction. We feel that short selling will simplify the calculations needed to predict the probability of a prediction occurring and will make our game less complicated, improving the accuracy of our results. Lastly, Google found correlations between the activities of workers sitting near each other or those with social relationships. Our Facebook app will similarly allow us to analyze the data and information exchanged in the context of a friend network.

Manski. Interpreting the Predictions of Prediction Markets. 2004. <http://www.nber.org/papers/w10359.pdf>

This paper discusses the theory of prediction markets. A prediction market determines probabilities that predictions will occur from the price that the shares are being traded at. However, the equilibrium price is not necessarily a reflection of every player’s subjective beliefs about the chance that the event will occur. Manski derives the mean belief probability E(qm) as lying within the range (pm2, 2pm - pm2) where pm is the market’s equilibrium price/probability assuming rational, risk-neutral players. Note that pm lies in the exact center of E(qm)’s range, but they are not necessarily going to be equal. Furthermore, the author notes that this equilibrium range will still hold after subjects update their beliefs. Lastly, Manski argues that the market price is NOT an accurate estimate of the likelihood of an event. That is, pm = P(qm > pm); the equilibrium price is the probability that a player’s subjective belief is larger than the price, such that “most persons have beliefs higher than the price when price is above 0.5, and most have beliefs lower than price when price is below 0.5”. Our practical experiment will be able to assess the validity of Manski’s theory in light of evidence that prediction markets are more accurate than experts and the efficient market hypothesis.

Kenneth J. Arrow, et.al. “The Promise of Prediction Markets”. Science Vol. 320, 16 May 2008
<http://www.arlingtoneconomics.com/studies/promise-of-prediction-markets.pdf>

“The Promise of Prediction Markets” provides a general overview of prediction markets, and evaluates their interactions with current gambling regulatory agencies. The paper stresses the power of prediction markets as a forecasting tool that is much more accurate and resistant to disturbances and outliers. Finally the author suggests a number of small legal reforms on the regulatory level that would allow research into prediction markets to continue without having to significantly reform gambling legislation.

The main obstacle to prediction markets legally is the fact that it is considered internet gambling, which is explicitly forbidden in eight states, and severely restricted on the federal level. While it is possible to get around these difficulties by setting up the markets overseas, such markets would not be as effective as domestic analogs due to the logistical difficulties of aggregating information from such a market. Instead, the author proposes a number of changes to the treatment of prediction markets within the United States. The first is to treat prediction markets similarly to futures markets (through the related regulatory agency, the CFTC) by issuing safe-harbor rules for small-stakes markets that operate for 1). nonprofit research purposes, 2). government research and planning purposes and 3). internal operation of private businesses. Within this scope, the authors argue that prediction markets should be allowed to price any economically meaningful event under any set of publicly available rules to stimulate fair use and research. Finally, the authors urge more permanent reform at the federal level that would separate prediction markets from gambling, allowing them to function as useful forecasting tools.

With this in mind, our goal is to implement a free prediction market game to help demonstrate the potential value in legalizing prediction markets in the United States.

Emile Servan-Schreiber, Justin Wolfers, David Pennock, Brian Galebach. “Prediction Markets: Does Money Matter?” Electronic Markets Vol. 14 No. 3
<http://bpp.wharton.upenn.edu/jwolfers/Papers/DoesMoneyMatter.pdf>

Servan-Schreiber et. al. conduct an experiment to test the improvement gained in the accuracy of prediction markets when the market is based on real money as opposed to virtual money. Surprisingly, over 208 experiments, there was no significant distinction between the accuracy of the virtual-money and real-money markets. The authors find that the accuracy of the prediction market was much more strongly determined by the knowledge and motivation of the user base rather than the presence of real money. While real money payoffs can be used to attract such a knowledgeable and motivated user base, a number of other reward schemes can be just as effective, such as community recognition, prizes awarded to best forecasters, and other general marketing reward systems.
The experiments used in this study were run on TradeSports (real money) and NewsFutures (virtual money), which are both sports trading exchanges targeted to the U.S. with large established user bases. While the markets had nearly identical forecasting performance, internal behavior did differ, suggesting that the two markets did not simply align to the same publicly available information.
The results of this paper are of particular interest in context of social network applications of prediction markets. In a social media environment, it is easy to provide non-monetary rewards in the market, which would greatly simplify the implementation of a prediction market, and avoid any legal difficulties discussed in the previous article. A possible extension of this project would be to compare the performance of the monetary market with one based on non-monetary rewards and attempt to verify the result of this study. For example, we could copy predictions that appear on InTrade and other sites and compare our accuracy.

10 week Timeline

App Launch for data gathering (Weeks 1-2)

Since we want to gather as much data as possible, we would like to focus our early efforts on producing the app and publicizing it early. Due to the importance of launching the app as soon as possible, we will focus our efforts early on in producing the app and publicizing it. We will develop the app in Google App Engine with an addition of a Facebook App that will interface with it. We plan on adding tweaks and features the app throughout the term to improve user experience and obtain additional data to analyze. For instance, one of the tweaks we are considering is posting a link to an article under every question we post to incentivize the users to do research on a topic before making a prediction.

Data collection (continuous)

After publishing the app for the wider audience to use, we plan to collect data of users’ usage statistics, e.g. questions answered, bid amounts, bid times and friend networks. Based on these we can infer additional information such as the users’ final prediction and its agreement with the actual result of the prediction question posted. With the timestamps on bids we can also perform temporal analysis of the data to infer prediction trends.

Prediction creation (continuous)

The main content of our app is creating prediction problems for the app to resolve. This task requires making questions that can be resolved in a relatively quick time frame, i.e. questions to which answers we can find out shortly after they were posted. We want to make as many predictions as possible throughout the term, since this is crucial to making our data sets as big as possible.

Setup scripts/functions to graph and analyze data (Weeks 3-5)

With the framework for collecting data already available, we plan on analyze the collected data. We wish to infer statistics that will allow us to quantitatively assess the accuracy of our prediction market. Secondly, we would like to analyze the temporal convergence of our system on predictions and answers. In order to accomplish these tasks we need to develop efficient data analysis script that will perform on our (hopefully) large data sets. Furthermore, we would like to setup some scripts to be prepared to analyze the data we obtain from the experiments in the following weeks.

Experiments (Weeks 6-8)

We will perform some insider information experiments. For instance, we would like to perform insider information experiments in which we observe prediction trends due to information disclosure to some selected groups of users on Facebook. This will require creating some incentive for the users’ to participate in the experiment, possibly with benefits such as in-game achievements. The second experiment we wish to perform is the market bubble experiment. By manipulating the market through buying or selling a lot of a single prediction, we would like to observe the behavior of the market. We would like to see whether introducing bias to the market would change the outcome of the prediction to see how resilient our system is to manipulation (if at all). If time allows, we will consider other experiments to perform, but the priority falls on performing the two aforementioned ones.

Information cascade analysis (Weeks 9-10)

The final thing we would like to analyze is the spread of information through friend networks. In our insider information experiment we would like to analyze the spread of prediction trends among users’ Facebook friend networks. Suppose user A was one of the users selected for the insider information experiment and therefore changes his prediction. Then, does his friend network react to the fact that the user A is one of the insiders or not? This is the question we are trying to meaningfully answer during this period.  Aside from experiments, we would also like to analyze trends in friend networks. Do tightly clustered friends exhibit similar properties while making predictions? To what extent do different social groups tend to make different predictions? We wish to infer characteristics such as these about the users’ friend networks.

End Product

The most important step, of course, is the actual application.  The Facebook game is, and needs to be, the first thing our team implements.  At a minimum, this means the user interface, the ability to buy and sell shares of questions, the ability to track the budgets of the users, and a list of yes/no questions.  There are also additional features we can add to this if we have time.  Keeping track of those with the best performance through a leaderboard system would incentivize users to do well, as would increasing their budgets based on performance.  Creating a system for accepting feedback would be helpful, as would allowing users to submit their own questions.  Additionally, we’ve considered analyzing the effects of a sort of insider trading, for example, discretely informing certain users that one of the coin flips in a hypothetical prediction came up heads.  In case time becomes an obstacle, of course, the application will go public once we’ve implemented the minimum necessary functions, and the optional features will be incorporated piecemeal as our schedules allow.

By the end of the project we hope to collect a great quantity of useful data.  Each transaction should record the price of that transaction, the time when the transaction went through, and the users involved.  We need to record the current price - and the prices of offers - of the predictions, although it may be better to collect this data in intervals rather than every transaction.  In addition, we will record the friend network of our users.  Analyzing the data is more complicated and heavily depends on the amount of data we’re able to collect.  Ideally, we’d like to test the accuracy of the prediction market, the time the market takes to converge on a final price, and how changes in the market cascade through the friend networks of the users.

The greatest obstacle to the project is obtaining a large enough quantity of data.  Getting enough people to play the games might be an issue; there are methods we can use to try to attract attention, but it is still possible that we just won’t have enough users to gather meaningful data on the cascading effects.  However, even if there aren’t enough users to analyze the cascading effects we should still be able to analyze the accuracy with which the users predicted the questions.  Additionally, analyzing the time it took the markets to converge on the final prices would be more difficult with fewer users, but it shouldn’t be impossible.

Time is also an obstacle.  Firstly, it will take some time to complete the application itself; we can’t even begin collecting data until it’s ready to go public on Facebook.  Secondly, getting enough users could take time; analyzing the cascading effect in particular requires a great many users to participate in the same prediction in order to get a good representation of the friend network.  Finally, the predictions themselves require time; not only do the users require time to converge on a final answer the individual games cannot end until after the corresponding real-world event comes to pass.  Fully analyzing the data could be time-consuming in itself, even if the application is completed on schedule and a sufficiently large number of users play the games. Even if we start running out of time, as long as we have the program itself and a bare minimal amount of users, we should be able to analyze the accuracy of the predictions, and time-to-convergence, but the cascading effects would be dropped.  

References

• http://en.wikipedia.org/wiki/Prediction_market
• http://www.intrade.com/v4/home/
• Cogwill et al. Using prediction markets to track information flows: evidence from Google. 2009. <http://bocowgill.com/GooglePredictionMarketPaper.pdf>
• Manski. Interpreting the Predictions of Prediction Markets. 2004. <http://www.nber.org/papers/w10359.pdf>
• Kenneth J. Arrow, et.al. “The Promise of Prediction Markets”. Science Vol. 320,

16 May 2008

<http://www.arlingtoneconomics.com/studies/promise-of-prediction-markets.pdf>

• Emile Servan-Schreiber, Justin Wolfers, David Pennock, Brian Galebach. “Prediction Markets: Does Money Matter?” Electronic Markets Vol. 14 No. 3

<http://bpp.wharton.upenn.edu/jwolfers/Papers/DoesMoneyMatter.pdf>