ICML Position Track: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit System [D]

The Problem with Current Review Systems

"Maybe the real AGI was the friends we made along the way" is a sentiment that always hits me, and conferences are the places where I reunite with old friends and meet new ones. However, when it comes to the submission/review experience, it might not be much of an exaggeration to say that almost everyone has many unpleasant experiences to share.

So I wrote a position paper to discuss this. I argue that current conference organizers lack proper tools to instill accountability and incentives for reviewers, authors, ACs, SACs, and others. The result is that undesired behaviors (e.g., lack of engagement) often go unchecked, while good behaviors are rarely rewarded and therefore don't happen. Honestly, when was the last time you witnessed any constructive internal discussion among reviewers or ACs?

And this won't change by writing nice words in Reviewer Guidelines or issuing a few desk rejections.

A Proposed Credit System

I propose a CREDIT SYSTEM where community members earn points by "doing good":

• Reviewing a paper would get you +1 point
• Being outstanding gets you +3 points

Members can then spend points to redeem perks ranging from traditional ones already adopted in current ML conferences (e.g., free registration) to new ones, such as requesting an additional reviewer to sort through a muddy situation.

Explorative Ideas Supported by the System

Such a system could also support explorative ideas like:

• Refundable submission fees: say 10 points per submission, which are then refunded regardless of acceptance, unless the submission is uniformly voted to be unready or ultra-low quality.
• Mobilizing non-author reviewers: non-author reviewers don't have the bandwidth issue of wearing both the author and reviewer hats and are not influenced by their own submissions.

And many more...

Closing Thoughts

My proposed system is far from perfect, but I'd like to think it takes a step toward a better conference review mechanism. I am also glad to see the position paper track becoming a welcoming platform for researchers to hash out their proposals and build toward a better future (see other review-related position papers below.)

For a topic that affects literally everyone at ICML, I am eager to hear your thoughts.