You know that One Ring–Thesis analogy? #Submitting

As elaborated by this tear-jerking, kinda accurate write-up?

Well here’s my One Ring

which I shall cast into Mount Doom on Friday.

I sincerely hope there will be no Gollum nearby to thwart me.

ERA Journal Ranking List Dropped

The Australian Research Council (ARC) has decided to remove ERA journal rankings in view of them “focusing ill-formed undesirable behavior in the management of research”.

(Although I’m guessing people here will just pretend they never read this and do nothing about it until 3 years later.)

[Innovation, Industry, Science and Research Minister Kim Carr] chastised the research community, saying: “There is clear and consistent evidence that the rankings were being deployed inappropriately within some quarters of the sector, in ways that could produce harmful outcomes, and based on a poor understanding of the actual role of the rankings.

“One common example was the setting of targets for publication in A and A* journals by institutional research managers.

“In light of these two factors — that ERA could work perfectly well without the rankings, and that their existence was focussing ill-informed undesirable behaviour in the management of research — I have made the decision to remove the rankings, based on the ARC’s expert advice.”

Australian Academy of Science Secretary for Science Policy Professor Bob Williamson has this to say:

“In our recent submission to the Australian Research Council, the Australian Academy of Science argued strongly that key areas such as interdisciplinary research and new research were seriously disadvantaged by journal ranking,” Professor Williamson said.

“This affected not only areas of science and technology, but also interactions between the sciences and the humanities.”

“It has been very distressing to see some universities using publications in highly ranked journals as the basis for funding, promotions, and even staff appointments.

“The ranking of a journal as A* does not mean every paper in it is first rate, and some very good papers may appear in smaller journals.

So how would the new ERA initiative include as changes? One proposed measure is to move more towards a peer review system. More comments from various experts here: interesting reads.

CICLing 2011: Sound Bites

I just don’t have time for long posts these days T_T so here goes: a bulleted list of sound-bites.

Jun’ichi Tsujii

• As engineers, you’d want to create applications that just work (so stats-based systems fit the bill nicely).
• As scientists though, you’d want to know when and why things don’t work sometimes, and is it worth your while to try handle these special (if they are) cases at all.
• And for this purpose, stats-based models can be like black-boxes that you can’t really inspect… and this is where computational linguistics can provide insights.

Hans Uszkoreit

• Don’t believe in the F₁ score blindly. For some applications you really just want the precision; for others recall is much more important.
• Example: for a system detecting possible reports of terrorist activities, a 1.0 recall means much more than 0.01 precision, because you can certainly afford a human perusing 100 reports and confirming that 1 genuine terrorist report.
• Unfortunately many inexperienced journal reviewers (Hans actually said that) would still insist that the system’s F₁ is too low and can’t be convinced of the more important role of recall in such cases.

Diana McCarthy

• Is word sense disambiguation still relevent especially when we talk about tagging with reference to a sense inventory? Because we increasingly feel this phenomena about words having a continuum of meanings rather than discrete ones (except in the case of homographs).
• I don’t have an answer for this though. Thoughts welcome!

(Thanks Prof McCarthy. I hope my idea works! Even if only somewhat.)

Chris Manning

• No, we don’t have special funding for the Stanford NLP tools development. T_T
• Yes, it’s really hard to maintain the code and projects as people leave. T_T
• Yes, it’d be nice if we had a long-term development team to take care of the software engineering and maintenance stuff so that the research students can concentrate on doing research, but this is really hard (impossible?) in reality. T_T

(Prof Manning… what can I say except that I/we feel the pain too? T_T)

QS sacked from The Times Higher Education World University Rankings

Phil Baty, current editor of the Times Higher Education (THE) World University Rankings and deputy editor of Times Higher Education magazine, confessed that the rankings published for the past six years using data supplied and owned by QS, are not fit for purpose. Baty said QS achieved only a tiny number of respondents to the “peer review” criterion, which made up 40% of a university's score. QS also did not take into account different citation habits between disciplines, thereby greatly handicapping the arts, humanities and social sciences.

THE has therefore ended their contract with QS, and enlisted Thomson Reuters instead. They're also looking at a major (if not complete) overhaul of the methodology to produce the 2010 Rankings, to be published this autumn.

(suspenseful, brooding music)

Hmmmm.

I shall reserve further comments until we see the resultant rankings. ;-)