 |
 |
Report of ACM Multimedia Conference Review Committee
Submitted by editor3 on April 2, 2011 - 05:04
March 2011
Highlights of Key Review Recommendations
1. Committee In December 2009, ACM SIGMM Director of Conferences (Mohan Kankanhalli) formed a new committee to look into the organization and review of technical papers for ACM Multimedia and other conferences under the umbrella of ACM SIGMM.
2. Terms of Reference The committee will review two aspects of ACM Multimedia and related conferences:
The Committee will only focus on the ACM Multimedia conference in this report. 3. Conference Organization 3.1 Establishment of the ACM Multimedia Conference Steering Committee (AMCSC) We recommend that an ACM Multimedia Conference Steering Committee (AMCSC) be formed to oversee the planning and organization of future SIGMM conferences, including ACM Multimedia. The roles of AMCSC are to oversee the proper selection of key appointment holders and smooth execution for each conference, as well as to ensure the adherence of best practices as prescribed in the conference guidelines. The AMCSC shall be chaired by the ACM SIGMM Director of Conferences, with members coming from Chairs of recent ACM SIGMM Conferences. Our recommendation is for AMCSC to have five more members, to be selected from the main organizers of the three major ACM SIGMM conferences, including the ACM Multimedia. The maximum period of membership for each member is two terms of two years, with one-third of members being retired every two years. 3.2 Co-Chairs of the Organization and Technical Program Committee These are important positions that represent the face and provide direction to the community. They should be filled by people who are well respected by the community, and who have strong interest in multimedia and are active participants of the conference. One important gauge of the maturity of a conference is to look at the number of past chairs who are still active participants of the conference. This is in comparison with other equivalent top conferences such as ACM SIGIR, CVPR or ICCV. The ACM Multimedia Conference Steering Committee (AMCSC) should take active role in the selection and appointment of these positions for the SIGMM conferences. The General Chairs (and Technical Program Co-Chairs) should preferably not be involved again in same capacity with future ACM Multimedia conferences within the next four years. This is to allow for grooming of new leaders for the community. The General co-Chairs need to make sure that program co-chairs for different tracks are working towards coordinated and relatively uniform milestones in the review process to avoid excessive delay of review procedure in any of the tracks. Mitigation measures need to be designed and properly taken by the Technical Program co-Chairs to avoid unexpected consequences during the review process. 3.3 Workshops The role of workshops is to offer a forum for discussions of emerging and specialized topic, and should remain so. The topics of workshop should be distinctive from the focuses of the main conference. In general, workshops should be used to nurture and grow new and emerging areas that do not currently have large participations. Once the topic of a workshop has matured, it should be transferred as a new area into the main conference. Given the above role, we recommend that the workshop should not be allowed to run for more than 3 consecutive years, and there should be no permanent "standing" workshops. The workshops should be selected from competitive proposals with an acceptance rate of 50% or less and, under normal circumstance. There should not be more than TEN (10) workshops per year. 3.4 Conference Registration and Participation The aim of the conference is to encourage full and well-rounded conference experience for as many people as possible. Package Registration:
Student Participations:
3.5 The Doctoral Symposium Program The SIG Multimedia executive committee and the conference will work towards mentoring and training of the next generation of multimedia researchers. The current doctoral symposium program at the conference suffers from weak participation, partly because it is considered to be less prestigious by many faculty members who mentor the students. They are thus reluctant to send their students to attend only this part of the program. The committee recommends that each year, the doctoral symposium program be tied to the SIG Multimedia dissertation award. We recommend that the top three candidates be invited to present their work. This can be presented as a special session, with the award being given at the main awards ceremony. If possible, the travel and boarding expenses for the three students should be covered by the conference. 3.6 Archival of Conference Experience Archives record past practices and experience, and provide valuable institutional memory to enable us to have a good understanding and maintenance of the best practices. We should archive what is important, including, organizing committee, PC committee, participation list with number of participants in all activities, technical program, budget, sponsorship leads, and lessons learnt, etc. In addition, video tapings of key note talks, award presentations, and other memorable conference activities should be archived too. The "lessons learnt" and past best practices should be integrated into the "conference guidelines", to be maintained by SIGMM. SIGMM should ensure that the conference chairs provide proper archival of key information within a certain period after the conclusion of the conference. The Archives could be made available to all SIGMMM members. Other Implementation Details: 3.7 Conference Venue and Social Media Experience The venue of the Conference needs to be carefully selected based on a number of criteria in order to make the conference a great meeting experience of the community. In particular, the cost of attending conference should not be so prohibitive that it discourages participation by students. Preferably, conferences should be held in a university, where the cost is generally lower and the atmosphere is more conducive to technical conference as compared to top business hotels. The conference should provide good Internet access to all participants, as well as fast and reliable Internet access for technical presentations and demonstrations. Conference organizers are expected to provide social media experience for attendees such as Facebook, Twitter etc, where the attendees can share comments regarding technical paper presentations and demos, etc. This will be especially helpful given that the question-answering times are rather limited after the paper presentations. These social media blogs will be done through a conference registration id to avoid flames and spams. 3.8 Food & Beverage Conference lunch is an important and integral part of the conference and should be provided as part of the conference participation. Such lunches allow people to meet and network in a neutral environment. Without providing lunches, people tend to gather among their close groups to look for places for lunch. All conference activities should be co-located in a main common area for teas and gatherings in order to ensure coherent conference atmosphere. Drinks, tea and coffee should be available at all times of the conference. This avoids long queues during the coffee breaks and provides a conducive environment for those who want to continue their discussions even after the breaks. The conference should try to integrate rather than segregate those with special dietary requirements, such as the vegetarians. 3.9 Student Volunteer Program With the growing size of conference, local organizers might need to draw on the help of student volunteers to support the conference. Hence, the conference should introduce a student volunteer program as with other conferences such as SIGCHI. Student volunteers will receive free conference registration and in return they will commit to perform a certain hours of services to the conference. Student volunteer program will also help to encourage greater student participation and integration, allowing local and foreign students to form valuable links for the benefit of the community in the long run. 4. Review Process of Technical Papers 4.1 Paper Review System The renewal of topics and emphasis of the conference should be done in a more systematic manner, with greater involvement of the ACM Multimedia Conference Steering Committee. ACM Multimedia should break with the current track-based system and adopt the area- based system. Instead of 4 broad tracks, which tend to be overlapping of often confusing to authors and reviewers, we should adopt, say, 10 or more focused areas each year. Each area will cover several closely related topics. It permits more flexibility in renewal of areas each year, and allows the submission and review of papers in each area to be more focused. As a rule, we should actively look into growing one or more new areas each year, while consolidating existing areas. This includes dropping areas from previous years that have become less active with lower importance. The conference chairs could call for and decide on new areas to be included for each year just before the call-for-papers is published. This process will involve the community in shaping the future of ACM Multimedia conferences. There should be better coordination between the technical program committees of the main conference and the workshops on what new areas are to be introduced and in what forum. In general, new or emerging areas should be started as workshops before being integrated into the main conference. 4.2 Acceptance of More Quality Papers Many ACM conferences are experiencing greater competition in paper acceptance, along with general increase in the number of researchers and submissions in many fields, including multimedia. We therefore recommend that ACM Multimedia Conference should accept more "quality" papers, raising from the current guideline of 15-17% to around 20%. It will signal to the community that we are now adopting a more inclusive approach rather than an exclusive attitude to accepting quality works. It will help to reduce some randomness in paper acceptance among the high quality papers when the acceptance ratio is too low (at around 15%), and there will be fewer unhappy authors. Higher acceptance, if done with good quality control, will also allow for a greater variety of good papers to be presented, and hence help the area to grow. Most importantly, by accepting more papers, it will go towards reducing the number of good (but rejected) papers in circulation in the conference system. It will alleviate the growth in the number of paper submitted to all the related conferences, and hence reduce the overall review loads for researchers in the field. In this respect, ACM MM should take the lead. 4.3 Structure of Technical Program Committee (TPC) 4.3.1 Two-Tier Technical Program Committee The adoption of area-based system means the implementation of 2-tier TPC, with the top- tier committee comprises Senior PC members (or Area Chairs) covering key areas to be emphasized in the conference; and a second tier committee (TPC Members) responsible for reviewing the papers in each areas. It is compulsory for Area Chairs to attend the TPC meeting. 4.3.2 Qualification of TPC members Area Chairs should be established and active members of the community, with good publication records in the conference and in-depth understanding of the review process. Regular TPC members should be active participants of the conference, who preferably have published ACM Multimedia papers before. This is where it is good to involve young researchers into the paper review process. Balance of the PC: We should try to achieve a good balance among the TPC members with regards to the community, expertise, age, gender and nationality. 4.3.3 Roles of TPC members Area Chairs recommend and recruit normal TPC members for each area; play mentoring role; ensure proper, thorough and timely review of each paper; summarize review of paper; and present the key aspects of papers at TPC meeting. Area Chairs do not need to read all papers, but they must have good understanding of the merits of all papers, especially those borderline papers. TPC members review the assigned papers and participate in the online discussion stage of paper review to help formulate the final decision for each paper. 4.3.4 Recruitment of PC members Area Chairs are recommended by the TPC and General co-Chairs, with the approval of the ACM Multimedia Conference Steering Committee. Depending on the projected number of submitted papers, each area will recruit one or more Area Chairs. TPC Members are recommended by a combination of TPC-co-Chairs and the Area Chair of respective areas. 4.4 Technical Contents of the Conference 4.4.1 Key Technical Areas to Focus The areas should cover all major areas to be emphasized and nurtured in the field. As a good practice, we should consider the introduction of one or more new areas (though some may be small initially) each year. 4.4.2 Variety of Technical Programs With the increased in number of submissions and level of participations, it is expected that the number of papers accepted, both long and short papers, will increase steadily over the years. While the primary mode of presentation for long papers has traditionally been oral, it has become increasingly infeasible for oral presentation with the increased number of long papers accepted, unless more days or more tracks are added to the conference. Depending on the content, levels of interest, constraints of conference facilities and suitability for oral presentation, TPC committee will make decisions on which accepted long papers are to be oral and which are to be presented as posters. This possibility should be made clear to potential authors in the call-for-papers notices. The committee feels that we should keep the current short paper track, which serves as a forum for papers describing work-in-progress or innovative new (breaking) ideas. It should not be viewed as a track for lower quality or rejected long papers. This view should be instituted in the review criteria. The TPC will also make recommendations on whether a good-quality rejected long paper should be considered for short paper session (as it describes new innovative ideas) or workshops (as it touches on areas relevant to respective workshops). For poster sessions (both long and short), it is recommended that an elevator pitch session should preferably be organized for authors to present their ideas within a short few- minute duration. This will allow the authors to present key ideas of their posters and audience to have quick overviews before deciding on which posters to explore in more details. Such sessions, which have increasingly becoming popular, also provide dynamism and life to the technical conference. 4.4.3 Relation with workshops One idea tried at ACM MM 2010 is that during the TPC meeting, the committee will recommend some of the good but rejected long papers to be redirected to "relevant" workshops for consideration. It is a good idea to have some good papers to be presented in more focused meetings and hence increase the coverage of the conference.
4.5 Paper Review Process 4.5.1 Anonymity in Review The norm of the review process should be double blind, where the reviewers and authors do not know each other. We should not switch back to the single-blind review system at the discretion of conference organizer as was done in the past. The practice of most top conferences is to disallow key decision makers such as the General co-Chairs, TPC co-Chairs, and Workshop co-Chairs to submit papers to their respective forums. This is to avoid conflicts of interests, both real and perceived. Since the conference is organized yearly, this should not present a big problem to key conference organizers. 4.5.2 Preferences/Bids Area Chairs and TPC members should be given the option to state their preferences on papers that they could and would like to review. TPC co-chairs and Area Chairs should work to ensure the relevance of papers submitted to respective areas, and reject those papers that are clearly out of scope or of low quality (say too short and are badly formatted etc) to reduce the overall review loads. 4.5.3 Number of reviews needed At least 3 reviews and a meta-review (to be done by the respective area chair) should be done for each paper. More reviews should be solicited if the recommendations are conflicting. 4.5.4 Level of details and quality of reviews: As the aim of the review is to constructively criticize and provide guidance to help the authors improve their papers, hence sufficient details should be provide in the review. We should disallow reviews that are one-liner or are too general. To help new reviewers, guidelines and samples of good reviews should be provided. For example, see the excellent guideline provided in SIGCHI. (http://chi2011.org/authors/chi-review- process.html) Length of the review must be substantial, and one approach is to prescribe a minimum of 200 words for review comments. The review form should offer different sections that help the reviewer structure the comments and area chairs evaluate the overall reviews. Hence the review form should include sections on: Summary, Strengths and Weaknesses of the Paper (say at least 3 points each), Novelty and Contributions to the Community; and Expertise and Confidence of the Reviewer, etc. 4.5.5 Review loads Area Chairs should handle 20 papers. As stated earlier, the role of area chairs is to manage the review process to ensure that all papers are properly reviewed and commented. They are not expected to read most of the papers. TPC members / reviewer should be expected to review up to 10 papers. 4.5.6 Rebuttal by authors on the reviews: This has become a common (best) practice in many top technical conferences, and many authors do expect the opportunity to comment on the reviews. The rebuttal should take place during the paper discussion period and before the face-to-face TPC meeting. 4.6 Conduct of TPC Meeting It should be compulsory for all the area chairs to attend the face-to-face or other form of synchronous TPC meetings. This ensures that all papers, especially the borderline papers, have someone who knows the contents of the paper well to argue for or against the paper. The "compulsory" attendance of TPC meeting should be made clear during the invitation stage. Only those who are able to attend the TPC meeting should agree to be area chairs. Optionally, as is done in the computer vision community, the conference may allocate budget to (partially) support the attendance of TPC meetings by the area chairs. This will make it easier for them to commit to attend the TPC meeting. One alternative is to organize the meeting in conjunction with one of the SIGMM conferences (such as the new ACM ICMR (International Conference on Multimedia Retrieval)), so as to facilitate the attendance of most area chairs. Face-to-face vs. synchronous online meeting o Face-to-face meeting:
4.7 Recognition of TPC Participations To encourage ownership in reviews, we further recommend that the name of Area Chair be appeared in the first page of the accepted paper handled by him/her. The conference should show appreciation and recognition to the best reviewers/PC members during the conference, through prizes presented during the banquet. 4.8 Keeping Track of Best practices Summary of what works and what needs to be improved. Handling of double submission - outright rejection of paper and banning the authors for 1 year (SIGIR's practice). Black-listing of bad reviewers. Rewards for good reviewers. |
|||
|
|
|||
|
|||
|
|
