Group Members:
Alex Fandrianto
Daniil Kitchaev
Tuan Anh Le
Richard O’Dowd
Robert Shimizu
Introduction
In this project, we intend to create a prediction market game on Facebook and analyze its convergence properties with respect to time, users, information, and other factors. We know prediction markets provide interesting insight into group perception of the probability of an event. So, by analyzing how the market converges, we will be able to see the accuracy and speed of group predictions. An advantage of using Facebook for this purpose is that we will have access to a large, ready supply of possible users, and an existing application system providing ease of use and reach for our game. In addition, Facebook allows us to explore the effects of social network structure as it relates to information cascade, predictions, and attempts to manipulate the results.
For this project, we intend to partially follow the example set by Intrade, a currently existing successful online prediction market, in constructing our prediction market. Each prediction will be a yes or no question, and be offered in a continuous double auction format. Individuals will be given a set number of points per day which they can use in all standard market activities, such as buying, selling, short-selling, etc. Shares will clear for zero points if false, and some fixed number of points if true, so individuals will have points frozen in their accounts to ensure that all potential losses are covered.
The data we will collect for the market will consist of price data as a function of time, participants, total number of bids, what the answer is, etc. In addition, we will construct a network graph containing user information regarding their bid prices, bid times, questions answered, etc. with linkages reflecting friends on Facebook. This network component is important as it will allow us to track information cascades, clustering of similar opinions, and other information based effects.
The first step of our data collection will be gathering data from normal predictions without any modifications on our part. The goal of this is to form a baseline for our further analysis. This will also allow us to get user feedback, increase our user base, and introduce any modifications that we will need for later runs. We will then follow this up with experiments as time allows, such as an insider information and market bubble experiment.
The insider information experiment entails introducing a prediction that we control, such as coin flips or dice rolls. We then feed this information to certain users and see how this affects convergence of the market. From prior knowledge, we expect to see direct information cascades from these individuals to others in their friends networks, and indirect information cascades as unrelated individuals react to market changes caused by these “insiders”. Thus this experiment will give us insight into how information cascades propagate throughout prediction markets, and how social networks affect information in prediction markets.
The market bubble experiment will test the robustness of prediction markets when faced with attempted manipulation. To achieve this end, we will artificially modify a prediction by either buying or selling a lot of bids, modifying the current perceived probability. This experiment will thus allow us to test how quickly prediction markets recover from manipulation if they do at all. In addition, this will give us more information regarding indirect information cascades and how they react to false or manipulative signals in prediction markets.
For analysis, there will be two types of data being analyzed: aggregate and user. As a first step for aggregate analysis, we will compile metrics related to accuracy, convergence speed, popularity, user bid clustering, etc. We can then compare this information to the results of the prediction and other predictive mechanisms to see the effectiveness of a prediction market. User data, on the other hand, will primarily be analyzed from a social network and information based view. From the user data, we will be able to see how bid choices propagate over social networks, and how they change with publicly available information such as bid price and private information such as insider trading. These two types of data will thus allow us to make useful inferences about the convergence properties of prediction markets in social networks.
Literature Review
Cogwill et al. Using prediction markets to track information flows: evidence from Google. 2009. <http://bocowgill.com/GooglePredictionMarketPaper.pdf>
Google carried out an internal prediction market with its employees. Questions varied from topics relevant to the company and topics that were of interest to employees but irrelevant to Google. The data collected revealed several biases that tended to decrease as employees gained experience with the market. One bias was favorite bias, where players would tend to overestimate predictions likely to come true (favorites) and underestimate the odds of predictions unlikely to pass (longshots). We hope to replicate these results. Players were also generally optimistic in predictions that favored Google, but this bias is unlikely to appear in our public prediction market. Another difference is our prediction market model. Google did not allow short selling of stock; instead allowing users to trade both positive and negative securities of a prediction. We feel that short selling will simplify the calculations needed to predict the probability of a prediction occurring and will make our game less complicated, improving the accuracy of our results. Lastly, Google found correlations between the activities of workers sitting near each other or those with social relationships. Our Facebook app will similarly allow us to analyze the data and information exchanged in the context of a friend network.
Manski. Interpreting the Predictions of Prediction Markets. 2004. <http://www.nber.org/papers/w10359.pdf>
This paper discusses the theory of prediction markets. A prediction market determines probabilities that predictions will occur from the price that the shares are being traded at. However, the equilibrium price is not necessarily a reflection of every player’s subjective beliefs about the chance that the event will occur. Manski derives the mean belief probability E(qm) as lying within the range (pm2, 2pm - pm2) where pm is the market’s equilibrium price/probability assuming rational, risk-neutral players. Note that pm lies in the exact center of E(qm)’s range, but they are not necessarily going to be equal. Furthermore, the author notes that this equilibrium range will still hold after subjects update their beliefs. Lastly, Manski argues that the market price is NOT an accurate estimate of the likelihood of an event. That is, pm = P(qm > pm); the equilibrium price is the probability that a player’s subjective belief is larger than the price, such that “most persons have beliefs higher than the price when price is above 0.5, and most have beliefs lower than price when price is below 0.5”. Our practical experiment will be able to assess the validity of Manski’s theory in light of evidence that prediction markets are more accurate than experts and the efficient market hypothesis.
Kenneth J. Arrow, et.al. “The Promise of Prediction Markets”. Science Vol. 320, 16 May 2008
<http://www.arlingtoneconomics.com/studies/promise-of-prediction-markets.pdf>
“The Promise of Prediction Markets” provides a general overview of prediction markets, and evaluates their interactions with current gambling regulatory agencies. The paper stresses the power of prediction markets as a forecasting tool that is much more accurate and resistant to disturbances and outliers. Finally the author suggests a number of small legal reforms on the regulatory level that would allow research into prediction markets to continue without having to significantly reform gambling legislation.
The main obstacle to prediction markets legally is the fact that it is considered internet gambling, which is explicitly forbidden in eight states, and severely restricted on the federal level. While it is possible to get around these difficulties by setting up the markets overseas, such markets would not be as effective as domestic analogs due to the logistical difficulties of aggregating information from such a market. Instead, the author proposes a number of changes to the treatment of prediction markets within the United States. The first is to treat prediction markets similarly to futures markets (through the related regulatory agency, the CFTC) by issuing safe-harbor rules for small-stakes markets that operate for 1). nonprofit research purposes, 2). government research and planning purposes and 3). internal operation of private businesses. Within this scope, the authors argue that prediction markets should be allowed to price any economically meaningful event under any set of publicly available rules to stimulate fair use and research. Finally, the authors urge more permanent reform at the federal level that would separate prediction markets from gambling, allowing them to function as useful forecasting tools.
With this in mind, our goal is to implement a free prediction market game to help demonstrate the potential value in legalizing prediction markets in the United States.
Emile Servan-Schreiber, Justin Wolfers, David Pennock, Brian Galebach. “Prediction Markets: Does Money Matter?” Electronic Markets Vol. 14 No. 3
<http://bpp.wharton.upenn.edu/jwolfers/Papers/DoesMoneyMatter.pdf>
Servan-Schreiber et. al. conduct an experiment to test the improvement gained in the accuracy of prediction markets when the market is based on real money as opposed to virtual money. Surprisingly, over 208 experiments, there was no significant distinction between the accuracy of the virtual-money and real-money markets. The authors find that the accuracy of the prediction market was much more strongly determined by the knowledge and motivation of the user base rather than the presence of real money. While real money payoffs can be used to attract such a knowledgeable and motivated user base, a number of other reward schemes can be just as effective, such as community recognition, prizes awarded to best forecasters, and other general marketing reward systems.
The experiments used in this study were run on TradeSports (real money) and NewsFutures (virtual money), which are both sports trading exchanges targeted to the U.S. with large established user bases. While the markets had nearly identical forecasting performance, internal behavior did differ, suggesting that the two markets did not simply align to the same publicly available information.
The results of this paper are of particular interest in context of social network applications of prediction markets. In a social media environment, it is easy to provide non-monetary rewards in the market, which would greatly simplify the implementation of a prediction market, and avoid any legal difficulties discussed in the previous article. A possible extension of this project would be to compare the performance of the monetary market with one based on non-monetary rewards and attempt to verify the result of this study. For example, we could copy predictions that appear on InTrade and other sites and compare our accuracy.
10 week Timeline
App Launch for data gathering (Weeks 1-2)
Since we want to gather as much data as possible, we would like to focus our early efforts on producing the app and publicizing it early. Due to the importance of launching the app as soon as possible, we will focus our efforts early on in producing the app and publicizing it. We will develop the app in Google App Engine with an addition of a Facebook App that will interface with it. We plan on adding tweaks and features the app throughout the term to improve user experience and obtain additional data to analyze. For instance, one of the tweaks we are considering is posting a link to an article under every question we post to incentivize the users to do research on a topic before making a prediction.
Data collection (continuous)
After publishing the app for the wider audience to use, we plan to collect data of users’ usage statistics, e.g. questions answered, bid amounts, bid times and friend networks. Based on these we can infer additional information such as the users’ final prediction and its agreement with the actual result of the prediction question posted. With the timestamps on bids we can also perform temporal analysis of the data to infer prediction trends.
Prediction creation (continuous)
The main content of our app is creating prediction problems for the app to resolve. This task requires making questions that can be resolved in a relatively quick time frame, i.e. questions to which answers we can find out shortly after they were posted. We want to make as many predictions as possible throughout the term, since this is crucial to making our data sets as big as possible.
Setup scripts/functions to graph and analyze data (Weeks 3-5)
With the framework for collecting data already available, we plan on analyze the collected data. We wish to infer statistics that will allow us to quantitatively assess the accuracy of our prediction market. Secondly, we would like to analyze the temporal convergence of our system on predictions and answers. In order to accomplish these tasks we need to develop efficient data analysis script that will perform on our (hopefully) large data sets. Furthermore, we would like to setup some scripts to be prepared to analyze the data we obtain from the experiments in the following weeks.
Experiments (Weeks 6-8)
We will perform some insider information experiments. For instance, we would like to perform insider information experiments in which we observe prediction trends due to information disclosure to some selected groups of users on Facebook. This will require creating some incentive for the users’ to participate in the experiment, possibly with benefits such as in-game achievements. The second experiment we wish to perform is the market bubble experiment. By manipulating the market through buying or selling a lot of a single prediction, we would like to observe the behavior of the market. We would like to see whether introducing bias to the market would change the outcome of the prediction to see how resilient our system is to manipulation (if at all). If time allows, we will consider other experiments to perform, but the priority falls on performing the two aforementioned ones.
Information cascade analysis (Weeks 9-10)
The final thing we would like to analyze is the spread of information through friend networks. In our insider information experiment we would like to analyze the spread of prediction trends among users’ Facebook friend networks. Suppose user A was one of the users selected for the insider information experiment and therefore changes his prediction. Then, does his friend network react to the fact that the user A is one of the insiders or not? This is the question we are trying to meaningfully answer during this period. Aside from experiments, we would also like to analyze trends in friend networks. Do tightly clustered friends exhibit similar properties while making predictions? To what extent do different social groups tend to make different predictions? We wish to infer characteristics such as these about the users’ friend networks.
End Product
The most important step, of course, is the actual application. The Facebook game is, and needs to be, the first thing our team implements. At a minimum, this means the user interface, the ability to buy and sell shares of questions, the ability to track the budgets of the users, and a list of yes/no questions. There are also additional features we can add to this if we have time. Keeping track of those with the best performance through a leaderboard system would incentivize users to do well, as would increasing their budgets based on performance. Creating a system for accepting feedback would be helpful, as would allowing users to submit their own questions. Additionally, we’ve considered analyzing the effects of a sort of insider trading, for example, discretely informing certain users that one of the coin flips in a hypothetical prediction came up heads. In case time becomes an obstacle, of course, the application will go public once we’ve implemented the minimum necessary functions, and the optional features will be incorporated piecemeal as our schedules allow.
By the end of the project we hope to collect a great quantity of useful data. Each transaction should record the price of that transaction, the time when the transaction went through, and the users involved. We need to record the current price - and the prices of offers - of the predictions, although it may be better to collect this data in intervals rather than every transaction. In addition, we will record the friend network of our users. Analyzing the data is more complicated and heavily depends on the amount of data we’re able to collect. Ideally, we’d like to test the accuracy of the prediction market, the time the market takes to converge on a final price, and how changes in the market cascade through the friend networks of the users.
The greatest obstacle to the project is obtaining a large enough quantity of data. Getting enough people to play the games might be an issue; there are methods we can use to try to attract attention, but it is still possible that we just won’t have enough users to gather meaningful data on the cascading effects. However, even if there aren’t enough users to analyze the cascading effects we should still be able to analyze the accuracy with which the users predicted the questions. Additionally, analyzing the time it took the markets to converge on the final prices would be more difficult with fewer users, but it shouldn’t be impossible.
Time is also an obstacle. Firstly, it will take some time to complete the application itself; we can’t even begin collecting data until it’s ready to go public on Facebook. Secondly, getting enough users could take time; analyzing the cascading effect in particular requires a great many users to participate in the same prediction in order to get a good representation of the friend network. Finally, the predictions themselves require time; not only do the users require time to converge on a final answer the individual games cannot end until after the corresponding real-world event comes to pass. Fully analyzing the data could be time-consuming in itself, even if the application is completed on schedule and a sufficiently large number of users play the games. Even if we start running out of time, as long as we have the program itself and a bare minimal amount of users, we should be able to analyze the accuracy of the predictions, and time-to-convergence, but the cascading effects would be dropped.
References
- http://en.wikipedia.org/wiki/Prediction_market
- http://www.intrade.com/v4/home/
- Cogwill et al. Using prediction markets to track information flows: evidence from Google. 2009. <http://bocowgill.com/GooglePredictionMarketPaper.pdf>
- Manski. Interpreting the Predictions of Prediction Markets. 2004. <http://www.nber.org/papers/w10359.pdf>
- Kenneth J. Arrow, et.al. “The Promise of Prediction Markets”. Science Vol. 320,
16 May 2008
- Emile Servan-Schreiber, Justin Wolfers, David Pennock, Brian Galebach. “Prediction Markets: Does Money Matter?” Electronic Markets Vol. 14 No. 3
No comments:
Post a Comment