Friday, April 1, 2011

New guidelines for the FOCS program committee

I am a member of the FOCS program committee this year, and today we received guidelines for our upcoming work. There are several noteworthy changes.

In particular, this year the program committee will produce, not merely a list of accepted papers, but a total ordering of accepted papers. This will greatly serve the field by giving each conference participant a quick way to choose which talks to attend, in areas outside their specialty. It will also give much more accurate information to put in people's vita. For this year, it is experimental only, but if it becomes mainstream, they will now be able to say, not merely: "I had 5 papers accepted at STOC/FOCS in the past four years", but can add: "... and the average rank of my accepted papers was 34.7". This will give researchers a powerful incentive to strive for quality work, by rewarding them for putting in the extra effort, in write up style and in generality of result, so as to make the difference between a borderline paper and a clear accept.

Moreover, the program committee is also requested to produce a ranking of the top 25 rejected submissions; however that ranking will only be sent to the authors confidentially; some may choose to keep it quiet, but others might be able to boast that "in the past four years, not only did I have 2 papers accepted at STOC/FOCS, but I also had 4 papers that almost made it, including the FOCS'11 best rejected paper." In particular, this may help PhD students in their job search, by showing their productivity. It eliminates the inherent unfairness of the 0-1 law coming from the accept-reject cutoff, and creates a relaxed version of "acceptance". As we know, fractional relaxations are much better behaved than integer problems.

I must say that I am a little bit surprised that such a significant change was not discussed at the business meeting beforehand. However, the guidelines made it clear that this is experimental only, and the attendees will be able to assess it at the next business meeting. In the worst case, I suppose that the program committee could decide to amend the guidelines and give a partial ordering only. If, on the other hand, this innovation is well received, since the information is also on file for the last several years, IEEE may also apply this change retroactively.

I am glad that our flagship conferences are always evolving and striving for improvements!

5 comments:

  1. I'm not on the PC, but I have heard rumours that actually the talks themselves will be ordered by the same total ordering. That way everybody will be there on Sunday morning and nobody has to hang around until Tuesday afternoon.

    ReplyDelete
  2. That sounds like an incredibly tough job. How can one come up with such a fine grading of paper quality? I could imagine a finer grading than accept/reject, but not much finer than best-paper(s)/accept-tier1/acecpt-tier2/reject. Good luck!

    ReplyDelete
  3. Thanks Cora, we'll do our best. I'm sure every one realizes that the ranking will have errors, of course, but it will still contain some valuable information.

    ReplyDelete
  4. Nick: the talk ordering that you are talking about has already been implemented since 2007. Didn't you know?

    ReplyDelete