Measure is unceasing

Forecasting Newsletter for November and December 2022

Highlights

Index

Browse past newsletters here (a), or view this newsletter on the EA forum here. If you have a content suggestion or want to reach out, you can leave a comment or find me on Twitter.

Prediction Markets and Forecasting Platforms

PredictIt

Racing against the clock, PredictIt is trying to get permission from the CFTC to continue existing. From PredictIt’s newsletter:

As you may know, the Commodity Futures Trading Commission (CFTC) decided in August to revoke the authorization that allowed PredictIt to operate effective Feb. 15, 2023 (a) (original CFTC source (a)). More than a dozen of your fellow PredictIt traders, as well as academic data users and PredictIt’s parent company Aristotle, have challenged this decision in federal court with the goal of keeping election markets open and operating past the CFTC’s arbitrary Feb. 15, 2023, deadline.

Given the time-sensitive nature of this fight, the plaintiffs filed a motion (a) last week to move the case to the United States Court of Appeals for the Fifth Circuit, which is the quickest path towards attaining justice for our traders and academic partners. We understand that legal battles do not generally make for exhilarating newsletter content, but as valued members of our community, we want you to know that we’re fighting on every available battlefield for the future of PredictIt and political prediction markets.

To that end, Aristotle, has also applied to the CFTC for permission to operate a Designated Contract Market (DCM), which will offer event contracts that include certain political outcomes. We’re well along in that process and hope to have approval in the coming months.

If approved, PredictIt Exchange will list many of the same contracts that you love: polling averages, nominations, confirmations and legislative outcomes, among other topics.

This next step toward becoming a regulated entity has been a long-time goal of the PredictIt team. We’ll be in touch soon with more information about how to sign up for PredictIt Exchange and other exciting developments in the works for next year. Stay tuned!

American Civics Exchange

The newly created American Civics Exchange is offering over-the-counter betting (a) on political events.

I’m not clear on how this is legal (a), but readers might consult their “What is ACE otc’s regulatory framework?” section in their FAQ (a).

The man behind the thing is Filip Pidot (a), who previously ran the Old Bull TV (a) youtube channel, and is generally considered a respectable community member.

Polymarket

Polymarket deploys an adapter (a) between its Keynesian beauty contest resolution mechanism, UMA (a), and actual resolutions. There is an “emergencyResolve” feature, which judging by the name could be used to override or bypass the UMA resolution. This is marginally concerning, because it could be used to siphon money away from markets, by betting on the less likely outcome and then resolving to it.

Image of the Polymarket-UMA adapter

Ultimately, I don’t think this matters very much, because if Polymarket was compromised, users' wallets could probably be drained through some change in the webpage. Still, this downgrades UMA from its previous status as a source of truth to merely a convenient resolution mechanism,

Insight Prediction

Insight Prediction introduced (a) some innovations around “sudden death markets”. It’s difficult to do market-making for markets about discrete events, because if the event happens, traders will quickly make bets and rake in profit before the market-maker notices. And so:

what we’ve implemented is the ability to have markets where we return all bets after a particular event has transpired. Bets placed before the event will be honored, while bets placed after will be returned. This of course requires bets in the market to be locked (72 hours)

You can see this functionality in this market: Will Elon Musk Father Another Child Before August 2023? (a) market.

Kalshi

Kalshi updates their API (a) and, separately, gets lampooned in Hacker News (a):

That’s neat but how does one hedge a $800k loan with a max position of $25k? Idea seems spurious. The difference is substantially higher. Compare a pre-hike 2.8% with a present 6.8%. Your $25k hedge is going to be pointless.

Before Kalshi, markets that allowed you to trade on economically relevant events were illegal or unregulated.

Totally untrue. PredictIt was legal and regulated and well-loved for many years, operating with the permission of the CFTC under a no-action letter. Then Kalshi hired former CFTC commissioner Brian Quintenz and soon, PredictIt was suddenly deemed by the CFTC to have committed still-unenumerated violations of the no-action letter.

This is a rotten way to do business and shame on YC for being involved in it.

INFER

INFER has a few quarterly reports (a) on their forecasts on Synthetic Biology, AI and Microelectronics. It also has a $5.25k (a) tournament.

When Metaculus got a $5.5M grant (a) earlier this year, I could mostly see this reverberating, them hiring more people, a few teams being created, people I was keeping tabs on getting hired, more forecasting contests happening, and so on. You know, $5M is a small fortune, and I try to listen to the pulse of these things. Manifold Markets got built on top of $1M.

So for a while now, it has been very unclear to me where the $8M that INFER’s parent organization got from Open Philanthropy (a) was going. Things just weren’t very visibly different from what I would expect to see if that grant hadn’t really happened. INFER’s forecasting rewards continued to be pretty small.

Taking a page from Andrew Gelman (a), who is extremely meek when pointing these kinds of issues out, there could be many explanations. For example, it’s possible that it does add up, that infrastructure and putting these kinds of reports out does take that much money. Or that they are spending the money slowly and sustainably for the long term. Or that the University of Maryland administration takes a really large chunk of funds. I am also assuming that the $8M in two years for two platforms doesn’t correspond to something in the very rough neighbourhood of $2M per year per platform, but it could be that that’s not the case. It’s also possible that the INFER forecasting sphere and the sphere that I pay the most attention to don’t overlap as much, which could be a positive. Still, the difference between the funds they receive and the impact that I perceive as for a while a small mystery to me.

Now, in its yearly review, INFER mentions (a) working on “an internal U.S. Government-wide site” for 2023. That does seem like it could absorb large amounts of funds. Still, personally, I’d love to see a bill of how much was spent where.

Metaculus

Metaculus held a Million Predictions Hackathon (a), and posted their year in review (a). They are also hosting a beginner tournament (a), which seems like a neat way of introduction to forecasting.

Manifold

Vincent Luczkow analyzes (a) whether Manifold Markets are well calibrated, and finds that they are:

manifold calibration chart

A Manifold Community member analyzed the performance of various prediction platforms on the US elections (a). I find that these analyses don’t really take into account that political outcomes are massively correlated. And so in years in which Democrats do well, platforms with a Democratic bias will do well, and vice-versa. Still, this isn’t totally uninformative.

Manifold adds swipe functionality to bet. I think this is pure evil genius, in the sense of being a very addictive skinner box, and I’m very impressed. You can try it here (a).

Manifold now has Android (a) and iOS (a) apps.

Odds and ends

The FTX Foundation, a major founder of innovative forecasting projects, exists no longer after the downfall of FTX. More on this in the next edition of this newsletter.

The Effective Altruism Forum has started a subforum for forecasting and estimation (a). Right now it mostly has some seed discussions started by employees of the Center for Effective Altruism. 

I don’t recommend that subforum for general forecasting discussion because I think that the EA Forum has too onerous “be nice” rules, whereas I tend to most like spaces that are maximally uncensorious and are willing to sacrifice other values in the pursuit of accuracy. For instance, Discord spaces from Metaculus, Polymarket & others have tended to be extremely unmoderated. 

Some comments from Good Judgment Open from the last long while:

A decentralized sports prediction market has gotten circa $5M (a) in funding (a).

Hypermind continues to host its Russia Punished (a) series of questions.

Scott Alexander wrote a Prediction Market FAQ (a) and is hosting a prediction contest (a) with a few days left to enter. If someone calculates expected reward over expected time, please post it as a comment.

Web3 betting platform loses $0.4M in hack (a).

Against the Odds (a) is a platform available to Spaniards which offers the opportunity to earn around $500/month through “crossed bets”, i.e., betting one thing in one prediction market and the opposite in another, in order to make a guaranteed profit. The service itself seems legit and comes recommended by a friend. But interested Spaniards should calculate whether this is worth it, e.g,. earning 7€ for 5 mins of work is 84€/hour, and in practice something like 50€/hour including time to set up things.

Forecasting community member Avraham Eisenberg was both despised and admired after pulling off a $114M heist (a) on a prediction platform. Now, he has been arrested in Puerto Rico (a), and will be facing trial. He remains 47th on the Metaculus leaderboard, as can be seen on his profile (a).

Opportunities

The Forecasting Research Institute (a) is a new think tank which has some senior research talent. Their research page (a) should give readers a good taste of what they are about; my impression is that they have a strong academic component and will be aiming to be a bit more institutional than other efforts in the space. I hesitate to mention this, but Phil Tetlock takes the perhaps somewhat ceremonial or advisory role of President.

Here (a) is a post on the EA forum introducing them. They are currently hiring (a) for research analyst ($70k/year), data analyst ($70k), content editor ($60k) and research assistant ($15/hour) positions.

Other people who are hiring are

Writeups

Shortform

The US 2022 congressional elections happened. Some relevant pieces on that:

With regard to comparisons between platforms, I don’t think that those analyses are completely uninformative. But I’d be hesitant to draw conclusions from one electoral season, because election outcomes are correlated. So for instance it could be that PredictIt is biased in favour of Republicans, Metaculus is biased in favour of Democrats, and Democrats did better this time. 

In Tracking the money flows in forecasting (a), I post some rough monetary valuation estimates for a large number of forecasting or prediction market organizations. The most out-of-place platform there is INFER, as reviewed above.

This online presentation (a) gives a few examples of systems whose expected value is far apart from the most likely value. h/t Adam Marblestone (a).

Gelman points to a history of forecasting competitions (a).

Jaime Sevilla posts Some research ideas in forecasting (a).

The great Robin Hanson proposes clean politician bonds (a):

Assume you will vote for 1 of 2 candidates for US Mayor, Governor, or House of Repr. Only 1 has posted $1M bond payable to anyone who proves their misbehavior re lies, bribes, or other cheating. On average, what is chance you vote for this candidate?

This brief blogpost (a) looks at when the geometric average of returns is a misleading indicator.

Alvaro de Menard writes Forecasting Forecasting (a), a short article looking at his perspective on the future of forecasting. Curiously, the article is fairly obsequious towards my forecasting group, Samotsvety.

There is an AI risk tournament whose results should be coming up soon that had superforecasters look at the chance of existential risk tournament due to AI. Participant Ege Erdil writes (a):

i was one of the people who participated in the AI forecasting thing you cite here, and the experience was unfortunately very bad due to various reasons

i would strongly push back on the idea that the people selected were in any sense “top forecasters”

Lao Mein writes Human-level Diplomacy was my fire alarm (a).

Nikos Bosse is working on a database of baserates (a).

A newsletter from Cultivate Labs highlights an article on Crowd Forecasting and Effective Altruism (a).

Ben West looks at the rate of fraud in YCombinator startups (a), and finds that “1-2% of $100M+ YCombinator-backed companies have faced serious allegations of fraud”.

This old Washington Post article (a) shows the annual risk of breakup for married and unmarried couples:

Breakup rates through time for different groups

Cultivate Labs' CEO reviews (a) his company’s activities in 2022. In particular:

The INFER project run by UMD’s ARLIS will be announcing a co-sponsored forecasting tournament with a significant player in the AI space next month, and will also be formally launching a U.S. Government-wide forecasting platform in the spring.

The European Commission, an organ of the European Union, has a page with economic forecasts and trends (a). Its economic forecasts are here (a) and here (a). It would be a neat research project to track results and check whether they are calibrated. It would also be easy to propose possible improvements: use range forecasts instead of point estimates, and use very liquid markets if you really care.

Here are some forecasts on British tourism (a).

Pinterest has a flashy Pinterest Predicts (a) webpage (methodology explained here (a)). Pinterest is used by people to plan stuff, and Pinterest can rely on the law of straight lines to predict that trends that are growing will continue to do so.

Does (a measure of) CEO overconfidence affect management forecasting and subsequent earnings management? (a). A paper finds that it does.

The International Journal of Forecasting has a special issue (a) on the M5 competition.

The Wall Street Journal reports on how companies use uncertainty and rising energy prices as the justification for increasing price (a). You could take them at their word. On the other hand, in Spain, where food prices have risen a whole lot, some people view these as illegitimate cash grabs.

This paper (a) tries to forecast the burden on the Chinese health system by tracking a large sample of older people.

Amazon offers some automated ML forecasting (a), as does Oracle (a).

Longform

Jared Leibowich looks at Modeling the End of Monkeypox (a) in a new EA magazine, Asterisk (a).

This blogpost (a) won $20k from GiveWell’s “Change our Mind Contest.” It looks at how their analyses could be improved by considering the uncertainty in their estimates, and does this at some depth, providing examples and recommendations.

Prominent anti-vaccine personality Steve Kirsh offered $500k bets (a) on the result of a public debate on whether vaccines save lives or not. Rootclaim (a), a hardcore Bayesian analysis organization, put significant efforts into doing their homework, and accepted the bet (a). Then Kirsh dishonourably chickened out (a). A transcript of the negotiations can be found here (a).

The Swift Centre has a report on what would happen if Russia uses a nuclear weapon (a), and is accepting commissions for topics to forecasting. They can be contacted at info@swiftcentre.org.

Andrew Gelman points (a) to PosteriorDB (a) a database of posteriors.

Friend of the newsletter Pratik Chougule recently published American Universities in the Middle East and U.S. Foreign Policy (a). It seems oddly specific, so I’d instead recommend his older & cheaper How to Make Money from Political Predictions (a).

Michael Story writes about why he generally doesn’t recommend internal prediction markets or forecasting tournaments to organisations (a).

In the News

Walter Frick of Nonrival (a) writes Journalists wake up to the power of prediction markets (a), a short piece introducing the concept and resources to journalists.

The New York Times has an article (a) on “Forecasting the Future of Election Prediction Markets”. It has this nice tidbit:

PredictIt is something of a hybrid. It is a joint venture. One partner is Victoria University of Wellington, a New Zealand institution. The other is Aristotle, a for-profit American political consulting, compliance, data and software company, whose founder, John Aristotle Phillips, first gained national attention in 1976 as “the A-bomb kid” — a Princeton undergraduate who successfully designed an atomic bomb in a physics class project.

I also find it interesting how the NYT article mostly cites academics rather than, e.g., Sam Altman’s comments to the CFTC (a) or other interesting figures (a).

Vegas Slots Online has a short article CFTC Ready to Deny Political Betting Request From Kalshi as PredictIt Levies a Lawsuit (a).

El País, a major Spanish newspaper, directly embedded prediction markets (a) in its reporting on the US election.

Newsweek, on Marjorie Taylor Greene’s Chances of Losing Georgia Re-election (a).

Fast Company has a lengthy profile on “the wild subculture of traders who bet on elections” (a).

The Atlantic (a): “In the eight years since PredictIt began, online sports gambling went from being outlawed nearly everywhere in the country to a booming industry… In this context, the crackdown on political betting seems somewhat silly… As it turns out, on another site, Insight Prediction, you can bet on PredictIt’s survival. The price of Yes, as of this writing, is 10 cents. Even if the CFTC follows through on its initial decision to shut the site down, it’s hard not to feel that PredictIt has, in some more meaningful way, already won.”

The Financial Times reviews a few of the wackier $1M investments (a) FTX made, including Manifold Markets.

Vox reviews (a) its yearly numerical predictions and makes new ones.

Extra: The tax-evaded Nazi money which influences German politics and US/NATO relationships has opened a new forecasting platform.

A smallish company rises (a) to become a major German publisher during the Nazi times. After World War II, the now disgraced father gives control of the company to one of his surviving sons. The son manages the business competently, and greatly expands it. In an attempted tax evasion scheme, he divides the company’s stock into a small percentage of voting stock and a large percentage of non-voting stock which he gives to a private foundation—I think this is the same trick which failed for the Ford Foundation (a).

Except that this time, the tricksucceeds (a), and after the death of the founder, it allows the family to keep control of assets while minimizing tax exposure. But the logic of the scheme requires that the company give out as few dividends as possible—because they would then be trapped inside the foundation and be difficult to get out. Which is why Bertelsmann has greatly expanded and is now a $30B behemoth which owns Penguin Random House (a) as one of many items in an absurdly long list of assets (a). Amusingly, they also have the same person (a) represent the interest of the family and of the foundation.

Heimat wider Heimat cover
Cover of the “Heimat, wider Heimat” book, which translates to something like “Motherland, oh Motherland again”, published by Bertelsmann. Note dog whistle connotations of Heimat. The author, Gustav Schröer, is one of four authors that I could quickly find that were both published by Bertelsmann and signed the “Declaration of Allegiance of German Writers (a)” to Adolf Hitler. The others are Rudolf G. Binding, Otto Flake and Fritz Müller-Partenkirchen. The case of Otto Flake is particularly interesting, since Bertelsmann rescued him from poverty in 1958 by reprinting old books of his.

Now, the Bertelsmann foundation has made a small contribution to the forecasting world by starting the Range Forecasting (a) platform, which runs on Cultivate Labs software. Per a recent Cultivate Labs newsletter:

Earlier this year, the Bertelsmann Stiftung and Bertelsmann Foundation, in partnership with Cultivate Labs, launched RANGE (Rethinking Assumptions in a New Geostrategic Environment) to forecast on topics critical to the transatlantic relationship. RANGE asks questions related to systemic rivalry with China, climate change, democracy, geopolitics, trade, and economics, all to gain a greater perspective of the transatlantic relationship and to provide analysis and recommendations for U.S. and European policymakers. 

At a critical time for strengthening global relations, the Stiftung and Foundation aim to engage an international audience in understanding how the transatlantic relationship affects us all, and how it can and should be shaped for the future. Read our recent announcement about the partnership and learn more at rangeforecasting.org.    RANGE was featured this week at the 2022 Futures Forum in Münster, Germany, a side event of the G7 minister’s meeting, which brought together American and German experts and rising leaders from civil society, academia, the private sector, and government to discuss transatlantic relations and the intersection of democracy and technology.

Earlier today at the Forum, Cultivate Labs CEO Adam Siegel spoke alongside Bertelsmann Foundation Executive Director Irene Braam about RANGE and the ways that crowd forecasting can help us better understand and strengthen transatlantic relations.

I do think that this new forecasting platform might be of some use. And I want to express my continued high regard for Cultivate Labs as one of the few platforms that have the proven ability to bring forth a new platform into existence, and maintain it through time. But I guess that reputational laundering of billionaires' money is easier to notice when it is billionaires I don’t care much about. And in this particular case, the provenance of the money seems so illegitimate, so rotten, as to be almost comical.

Yearly housekeeping

Here (a) is my yearly feedback form for this newsletter. I care most about readers' thoughts on whether this newsletter should be posted once a month or once every two months, and about freeform thoughts.

Also note that as the price I assign to my time rises, I’m becoming more ambivalent about whether to continue publishing this newsletter. If you have strong feelings about whether this newsletter should continue existing, consider becoming a paid subscriber, because a) I consider that a strong signal that I should continue writing this newsletter, b) I like the money, and c) the rate of current paid subscriptions is a pitiful 1.6% (a).

Note to the future: All links are added automatically to the Internet Archive, using this tool (a). “(a)” for archived links was inspired by Milan Griffes (a), Andrew Zuckerman (a), and Alexey Guzey (a).

If I had to put my money where my mouth is, my mouth wouldn’t be very busy. I’m just sayin’… — Anonymous commenter (a)