.bib BibTeX file of all papers

NTCIR-6 Preface

NTCIR-6 Preface Noriko Kando, National Institute of Informatics.

• "CLEF: Ongoing Activities and Plans for the Future", Maristella Agosti (University of Padua), Giorgio Maria Di Nunzio (University of Padua), Nicola Ferro (University of Padua) and Carol Peters (ISTI-CNR). (Paper PDF) (Slides PDF)
• "What's New at TREC: Blog and Legal Discovery Search at TREC-2006", Ian Soboroff (NIST). (PDF)

"This past year, the Text REtrieval Conference (TREC) started two new tracks. One was the Blog track – given a large collection of blog posts and their comments, the task was to locate opinions about products, people, organizations, etc. The other new track was the Legal Track. This track seeks to build test collections for searches that occur during the discovery portion of a lawsuit. The Legal track’s collection contains seven million scanned documents, and as such presents the additional challenge of searching highly variable quality OCR output. "

• "Test collections for all", Mark Sanderson (University of Sheffield). (PDF)

"Researchers working in the IR field have placed much reliance on building test collections that can be used widely by many researchers. Many collections have been used for years even decades. In the age of contextual IR, this talk will advocate an alternative less tried approach, that of building many context specific collections, that don't require a great deal of effort to build but may not be all that re-usable. There is an increasing quantity of research examining this area and through an overview of this research, I shall argue that building your own collection is a better approach to take."

• "Overview of the Sixth NTCIR Workshop", Noriko Kando (National Institute of Informatics).(PDF) (Slides PDF)
• "Overview of CLIR Task at the Sixth NTCIR Workshop", Kazuaki Kishida ("School of Library and Information Science, Keio University"), Kuang-hua Chen ("Department of Library and Information Science, National Taiwan University"), Sukhoon Lee ("Department of Statistics, Chungnam National University"), Kazuko Kuriyama (Shirayuri College), Noriko Kando (National Institute of Informatics) and Hsin-Hsi Chen ("Department of Computer Science and Information Engineering, National Taiwan University"). (PDF)

"The purpose of this paper is to overview research efforts at the NTCIR-6 CLIR task, which is a project of large-scale retrieval experiments on cross-lingual information retrieval (CLIR) of Chinese, Japanese, Korean, and English. The project has three sub-tasks, multi-lingual IR (MLIR), bilingual IR (BLIR), and single language IR (SLIR), in which many research groups from ten countries or regions are participat-ing. This paper describes the system of the NTCIR-6 CLIR task and its test collection (document sets, topic sets, and method for relevance judgments), and reviews CLIR techniques used by participants and search performance of runs submitted for evaluation."

• "Using Unigram and Bigram Language Models for Monolingual and Cross-Language IR", Lixin Shi (University of Montreal) and Jian-Yun Nie (University of Montreal). (Paper PDF), (Slides PDF) (Poster PDF)

"Due to the lack of explicit word boundaries in Chinese, and Japanese, and to some extent in Korean, an additional problem in IR in these languages is to determine the appropriate indexing units. For CLIR with these languages, we also need to determine translation units. Both words and n-grams of characters have been used in IR in these languages; however, only words have been used as translation units in previous studies. In this paper, we compare the utilization of words and n-grams for both monolingual and cross-lingual IR in these languages. Our experiments show that Chinese character n-grams are reasonable alternative indexing and translation units to words, and they lead to retrieval effectiveness comparable to or higher than words. For Japanese and Korean IR, bi-grams or a combination of bigrams and unigrams produce the highest effectiveness."

• "IASL System for NTCIR-6 Korean-Chinese Cross-Language Information Retrieval", Yu-Chun Wang (Academia Sinica), Cheng-Wei Lee (Institute of Information Science, Academia Sinica/National Tsing-Hua University), Richar Tzong-Han Tsai (Academia Sinica) and Wen-Lian Hsu (Academia Sinica). (Official Paper PDF) (Revised Paper PDF 2007-05-29) (Slides PDF) (Poster PDF)

"This paper describes our Korean-Chinese cross-language information retrieval system for NTCIR-6. Our system uses a bilingual dictionary to perform query translation. We expand our bilingual dictionary by extracting words and their translations from the Wikipedia site, an online encyclopedia. To resolve the problem of translating Western people's names into Chinese, we propose a transliteration mapping method. We translate queries form Korean to Chinese by using a co-occurrence method. When evaluating on the NTCIR-6 test set, the performance of our system achieves a mean average precision (MAP) of 0.1392 (relax score) for title query type and 0.1274 (relax score) for description query type."

• "Improving translation accuracy in web-based translation extraction", Chengye Lu (Queensland University of Technology), Yue Xu (Queensland University of Technology) and Shlomo Geva (Queensland University of Technology). (PDF)

"In this paper, we present some approaches to improve translation accuracy in web-based translation extraction. In previous work, researchers used term extraction techniques that are proposed under large static corpus. We proposed some approaches that can improve the translation accuracy in web-based translation extraction which relies on small dynamic small corpus. We also analyzed the difference in using local text corpus and web corpus as disambiguation source."

• "Toshiba BRIDJE at NTCIR-6 CLIR: The Head/Lead Method and Graded Relevance Feedback", Tetsuya Sakai ("NewsWatch, Inc."), Makoto Koyama (Toshiba), Tatsuya Izuha (Toshiba), Akira Kumano (Toshiba), Toshihiko Manabe (Toshiba) and Tomoharu Kokubu (Toshiba). (PDF)

"At NTCIR-6 CLIR, Toshiba participated in the Monolingual and Bilingual IR tasks covering three topic languages (Japanese, English and Chinese) and one document language (Japanese). For Stage 1 (which is the usual ad hoc task using the new NTCIR- 6 topics), we submitted two DESCRIPTION runs and two TITLE runs for each topic language. Our first search strategy is Selective Sampling with Memory Resetting, and our second one is the Head/Lead method, which uses the Selective Sampling run as one of the components for data fusion. According to the Relaxed and Rigid Mean Average Precision statistics released by the organisers, we are the top performer in all six subtasks. For Stage 2 (which reused the NTCIR-3, 4 and 5 test collections), we repeated our two Stage 1 strategies in order to enable analysis across all four test collections. Moreover, we conducted some unofficial true relevance feedback experiments by exploiting the graded relevance data provided in the test collections. Our automatic run results show that the Head/Lead method slightly but consistently improves performance, while our unofficial ÒinteractiveÓ run results suggest that graded-relevance metrics favour graded relevance feedback while Average Precision favours binary relevance feedback. In addition, our significance tests suggest that the NTCIR-6 Japanese test collection is ÒharderÓ than previous collections."

• "POSTECH at NTCIR-6: Combining Evidences of Multiple Term Extractions for Mono-lingual and Cross-lingual Retrieval in Korean and Japanese", Seung-Hoon Na (POSTECH), Jungi Kim (POSTECH), Ye-Ha Lee (POSTECH) and Jong-Hyeok Lee (POSTECH). (Official Proceedings Paper PDF) (Revised Paper 2007-05-22)

"This paper describes our methodologies for NTCIR-6 CLIR involving Korean and Japanese, and reports the official result for Stage 1 and Stage 2. We participated in three tracks: K-K and J-J monolingual tracks and J-K cross-lingual tracks. As in the previous year, we focus on handling segmentation ambiguities in Asian languages. As a result, we prepared multiple term representations for documents and queries, of which ranked results are merged to generate final ranking. From official results, our methodology in Korean won the top in 6 subtasks of total 9 subtasks for Stage 2,and won the top in 2 subtasks of total 3 subtasks for Stage 1. Even though our system is the same as the previous one, final performances from NTCIR-3 to NTCIR-5 are further improved over our previous results by slightly modifying the feedback parameters."

• "Monolingual Experiments with Far-East Languages in NTCIR-6", Samir Abdou (University of Neuchatel) and Savoy Jacques (University of Neuchatel). (PDF)

"This paper describes our third participation in an evaluation campaign involving the Chinese, Japanese and Korean languages (NTCIR-6). Our participation is motivated by three objectives: 1) study the retrieval performances of various probabilistic and language models for these languages; 2) compare the relative retrieval effectiveness of a combined Òunigram & bigramÓ indexing scheme combined with an automatic word-segmenting approach for Chinese and Japanese languages; and 3) evaluate the relative performance of the various data fusion strategies used to combine separate result lists in order to enhance retrieval effectiveness."

• "AINLP at NTCIR-6: Evaluations for Multilingual and Cross-Lingual Information Retrieval", Chen-Hsin Cheng ("Department of Information Management, Huafan University, Taiwan, R.O.C."), Reuy-Jye Shue ("Department of Information Management, Huafan University, Taiwan, R.O.C."), Hung-Lin Lee ("Department of Information Management, Huafan University, Taiwan, R.O.C."), Shu-Yu Hsieh ("Department of Information Management, Huafan University, Taiwan, R.O.C."), Guann-Cyun Yeh ("Department of Information Management, Huafan University, Taiwan, R.O.C.") and Guo-Wei Bian ("Department of Information Management, Huafan University, Taiwan, R.O.C."). (PDF)

"In this paper, a multilingual cross-lingual in-formation retrieval (CLIR) system is presented and evaluated in NTCIR-6 project. We use the language-independent indexing technology to process the text collections of Chinese, Japanese, Korean, and English languages. Different machine translation systems are used to translate the queries for bilingual and multilingual CLIR. The experimental results are discussed to ana-lyze the performances of our system. The effective-ness of query translations for bilingual and multilin-gual CLIR is discussed. In the evaluations, the Eng-lish version of topics performed better CLIR results to retrieve the Korean text collections than the Chi-nese version did. However, the Chinese version of topics performed better cross-language information retrieval results to retrieve the Japanese text collec-tions than the English version did."

• "NTCIR-6 CLIR-J-J Experiments at Yahoo! Japan", Sumio Fujita (Yahoo Japan Corporation). (Paper PDF), (Slides PDF)

"This paper describes NTCIR-6 experiments of the CLIRJ- J task, i.e. Japanese monolingual retrieval subtask, at the Yahoo group, focusing on the parameter optimization in information retrieval (IR). Unlike regression approaches, we optimized parameters completely independent from retrieval models so that the optimized parameter set can illustrate the characteristics of the target test collections. We adopted the genetic algorithm (GA) as optimization tools and cross-validated with 4 test collections, namely NTCIR-3,4,5, and 6 CLIR-J-J."

• "Search Between Chinese and Japanese Text Collections", Fredric Gey ("University of California, Berkeley"). (PDF)

"For NTCIR Workshop 6 UC Berkeley participated in Phase 1 of the bilingual task of the CLIR track. Our focus was upon Japanese topic search against the Chinese News Document Collection and upon Chinese topic searches retrieving from Japanese News document collection. We performed search experiments to segment and use Chinese search topics directly as if they were Japanese topics and vice versa. We also utilized Machine Translation (MT) software between Japanese and Chinese, with English as a pivot language. While Chinese search without translation against Japanese documents performed credibly well for title only runs, the reverse (Japanese topic search of Chinese documents without translation) was poor. We are investigating the reasons."

• "Chinese Information Retrieval Based on Document Expansion", Tingting He ("Department of Computer Science, Huazhong Normal University, 430079, Wuhan"), Li Li ("Department of Computer Science, Huazhong Normal University, 430079, Wuhan"), Guozhong Qu ("Department of Computer Science, Huazhong Normal University, 430079, Wuhan") and Yong Zhang ("Department of Computer Science, Huazhong Normal University, 430079, Wuhan"). (PDF)

"This paper describes our work at the sixth NTCIR workshop on the subtasks of monolingual information retrieval (CLIR). This is the second time we have participated in NTCIR. We have used query expansion methods in NTCIR-5 with related term groups, and this time we use document expansion. The traditional information retrieval model has limitations on finding related documents since it simply checks the existence of query terms in documents without considering the context of documents. Now we retrieve documents by vector space model and cluster the top-n documents to re-ranking the result set. Experiments show that our method achieves an average 3.2% improvement comparing with the method we have used in NTCIR-5 that adopts query expansion."

• "ISCAS in CLIR at NTCIR-6: Experiments with MT and PRF", Rui-hong Huang (ISCAS), Le Sun (ISCAS), Jing Li (ISCAS), Long-xi Pan (ISCAS) and Junlin Zhang (ISCAS). (PDF)

"We participated in the English-Chinese cross-language information retrieval (CLIR) E-C tasks in NTCIR6. Considering the special feature of crossing two different languages in CLIR, our main concerns in our experiment are 1) to evaluate the appropriateness of MT as a means of query translation in CLIR, 2) to evalua- te the effect of feedback in retrieval model to the performance of CLIR which has been discussed in some papers. Besides, we 1) applied Chinese word segmenter with quite high precision to ens- ure an exact indexing, 2) applied language model with relevance feedback as our retrieval model."

• "OASIS at NTCIR-6: On-line Query Translation for Chinese-Japanese Cross-Lingual Information Retrieval", Vitaly Klyuev (University of Aizu). (PDF) (Poster PDF)

"This paper reports results of Chinese - Japanese CLIR experiments using on-line query translation techniques. Approaches to employ English as a pilot language and to utilize several on-line translation systems are introduced. They were tested on NTCIR - 3, 4, 5, and 6 collections. Proposed procedures can be helpful under certain circumstances."

• "NTCIR-6 Monolingual Chinese and English-Chinese Cross Language Retrieval Experiments using PIRCS", Kui-Lam Kwok ("Queens College, CUNY") and Norbert Dinstl ("Queens College, CUNY"). (PDF)

"In NTCIR-6, our Stage-1 results which consist of using old queries retrieving on a different old collection, were not official because of late submission. Stage-2 submissions, which consists of repeating previous experiments, were on time. These NTCIR-6 experiments were conducted as new without referring to any previous knowledge about the runs. Comparisons with old results however were less favorable for about half the runs. We traced this to the accidental use of an out-dated module which sets the Zipf high frequency threshold too low, and leads to too many high frequency terms being removed from a query. Some runs are new and not submitted previously by us. These include: 'title' queries for NTCIR-3 monolingual Chinese and English-Chinese CLIR, and the English-Chinese CLIR runs for NTCIR-4."

• "Applying Multiple Characteristics and Techniques in the NICT Information Retrieval System at NTCIR-6", Masaki Murata (National Institute of Information and Communications Technology), Jong-Hoon Oh (National Institute of Information and Communications Technology), Qing Ma ("National Institute of Information and Communications Technology, Ryukoku University") and Hitoshi Isahara (National Institute of Information and Communications Technology). (PDF)

"Our information retrieval system takes advantage of numerous characteristics of information and uses numerous sophisticated techniques. Robertson's 2-Poisson model and Rocchio's formula, both of which are known to be effective, are used in the system. Characteristics of newspapers such as locational information are used. We present our application of Fujita's method, where longer terms are used in retrieval by the system but de-emphasized relative to the emphasis on the shortest terms; this allows us to use both compound and single-word terms. The statistical test used in expanding queries through an automatic feedback process is described. The method gives us terms that have been statistically shown to be related to the top-ranked documents that were obtained in the first retrieval. We also used a numerical term, QIDF, which is an IDF term for queries. It decreases the scores for stop words that occur in many queries. It can be very useful for foreign languages for which we cannot determine stop words. Furthermore, we used web-based unknown word translation for bilingual information retrieval. We participated in two monolingual information retrieval tasks (Korean and Japanese) and five bilingual information retrieval tasks (Chinese-Japanese, English-Japanese, Japanese-Korean, Korean-Japanese, and English-Korean) in NTCIR-6. We obtained good results in all the tasks."

• "NTCIR-6 CLIR Experiments at Osaka Kyoiku University - Term Expansion Using Online Dictionaries and Weighting Score by Term Variety -", Takashi Sato (Osaka Kyoiku University). (PDF) (Poster PDF)

"This paper describes experimental results of J-J subtask of NTCIR-6 CLIR. We expanded query term using online dictionaries in a WEB. It was effective for some topics of which average precision was low. Probabilistic model were employed for scoring, and we modified this score multiplying by the number of varieties of query terms, also. In most cases this works well. Query term reduction should be considered if this modified scoring fails."

• "Using Wikipedia to Translate OOV Term on MLIR", Chen-Yu Su (CYUT), Tien-Chien Lin (CYUT) and Shih-Hung Wu (CYUT). (PDF) (Poster PDF) (Link to online Wikipedia-based Term Translation Tool)

"We deal with Chinese, Japanese and Korean multilingual information retrieval (MLIR) in NTCIR-6, and submit our results on the C-CJK-T and C-CJK-D subtask. In these runs, we adopt Dictionary-Based Approach to translate query terms. In addition to tradition dictionary, we incorporate the Wikipedia as a live dictionary. Keywords: MLIR, Wikipedia, OOV terms"

• "JustSystems in Japanese monolingual information retrieval at NTCIR-6", Tetsuya Tashiro (JustSystems). (PDF)

"At the NTCIR-6 workshop, JustSystems Corporation participated in the Cross-Language Retrieval Task (CLIR). We submitted results to the track of monolingual information retrieval (Japanese to Japanese). The major goal of our participation is to evaluate performance and robustness of phrasal indexing and phrase down weighting combined with Language Modeling retrieval model."

• "Sampling Precision to Depth 9000: Evaluation Experiments at NTCIR-6", Stephen Tomlinson (Open Text Corporation). (PDF)

"We describe evaluation experiments conducted by submitting retrieval runs for the Chinese, Japanese and Korean Single Language Information Retrieval subtasks of the Cross-Lingual Information Retrieval (CLIR) Task of the 6th NII Test Collection for IR Systems Workshop (NTCIR-6). We show that a Generalized Success@10 measure exposes a downside of the blind feedback technique that is overlooked by traditional ad hoc retrieval measures such as mean average precision, R-precision and Precision@10. Hence an important retrieval scenario, seeking just one item to answer a question, is not properly evaluated by the traditional ad hoc retrieval measures. Also, for each language, we submitted a one-percent subset of the first 9000 retrieved items to investigate the frequency of relevant items at deeper ranks than the official judging depth of 100. The results suggest that, on average, less than 60% of the relevant items for Chinese and less than 80% for Japanese are assessed."

• "On the Robustness of Document Re-Ranking Techniques: A Comparison of Label Propagation, KNN, and Relevance Feedback", Yuen-Hsien Tseng (National Taiwan Normal University), Chen-Yang Tsai (Fu Jen Catholic University) and Ze-Jing Chuang (WebGenie Information LTD.). (PDF)

"This paper describes our work at the sixth NTCIR workshop on the subtask of C-C single language information retrieval. We compared label propagation (LP), K-nearest neighboring (KNN), and relevance feedback (RF) for document re-ranking and found that RF is a more robust technique for performance improvement, while LP and KNN are sensitive to the choice and the number of relevant documents for successful document re-ranking."

• "NCU in Bilingual Information Retrieval Experiments at NTCIR-6", Yu-Chieh Wu (NCU at Taiwan), Kun-Chang Tsai (NCU at Taiwan) and Jie-Chi Yang (NCU at Taiwan). (PDF)

"In this paper, we present the mono-lingual and bilingual ad-hoc information retrieval experimental results at NTCIR-6. This year we compare two different word tokenization levels for indexing, namely, unigram, and overlapping bigram. The two famous information retrieval models, i.e., language model, and BM-25 were adopted in our study. In the mono-lingual results show that our method achieved the average most runs, while the overlapping bigrams were indexed. The unigram level of words did the almost poor results in all runs. In the bilingual retrieval tasks, we translate the queries through a well-known machine translation tool. The evaluation results of our method were also given in the tail of this paper."

• "Information Retrieval Using Label Propagation Based Ranking", Lingpeng Yang (Institute for Infocomm Research), Donghong Ji (Institute for Infocomm Research) and Yu Nie (Institute for Infocomm Research). (PDF)

"The I2R group participated in the cross-language retrieval task (CLIR) at the sixth NTCIR workshop (NTCIR 6). In this paper, we describe our approach on Chinese Single Language Information Retrieval (SLIR) task and English-Chinese Bilingual CLIR task (BLIR). We use both bi-grams and single Chinese characters as index units and use OKAPI BM25 as retrieval model. The initial retrieved documents are re-ranked before they are used to do standard query expansion. Our document re-ranking method is done by a label propagation-based semi-supervised learning algorithm to utilize the intrinsic structure underlying in the large document data. Since no labeled relevant or irrelevant documents are generally available in IR, our approach tries to extract some pseudo labeled documents from the ranking list of the initial retrieval. For pseudo relevant documents, we determine a cluster of documents from the top ones via cluster validation-based k-means clustering; for pseudo irrelevant ones, we pick a set of documents from the bottom ones. Then the ranking of the documents can be conducted via label propagation. For Chinese SLIR task, experiences show our method achieves 0.3097, 0.4013 mean average precision on T-only run (Title based) at rigid, relax relevant judgment and 0.3136, 0.4071 mean average precision on D-only run (short description based) at rigid, relax relevant judgment. For English-Chinese BLIR task, experiences show our method achieves 0.2013, 0.2931 mean average precision on T-only run at rigid, relax relevant judgment and 0.1911, 0.2804 mean average precision on D-only run at rigid, relax relevant judgment."

• "NTCIR-6 Experiments using Pattern Matched Translation Extraction", Dong Zhou (University of Nottingham), Mark Truran (Univeristy of Teesside), Tim Brailsford (University of Nottingham) and Helen Ashman (University of Nottingham). (PDF)

"This paper describes our experiment methods and results in the Sixth NTCIR Workshop Meeting on Evaluation of Information Access Technologies. We introduce a Pattern Matched Translation Extraction (PMTE) approach to the analysis of mixed-languages web pages, which makes use of pattern matching to automatically extract the translation pairs. The experiment results demonstrated the proposed method is effective when translating Out-of-Vocabulary (OOV) terms, a wellknown problem in fields of cross-language information retrieval (CLIR), question-answering (QA), machine translation (MT) and knowledge discovery (KD). We also report the experiment results of single-language information retrieval (SLIR) and illustrate the performance through different collections in STAGE 2 of NTCIR-6."

• "Overview of the NTCIR-6 Cross-Lingual Question Answering (CLQA) Task", Yutaka Sasaki (University of Manchester), Chuan-Jie Lin (National Taiwan Ocean University), Kuang-hua Chen (National Taiwan University) and Hsin-Hsi Chen (National Taiwan University). (Official Proceedings Paper PDF), (Revised Paper 2007-06-04)

"This paper describes an overview of the NTCIR-6 Cross-Lingual Question Answering (CLQA) Task, an evaluation campaign for Cross-Lingual Question Answering technology. In NTCIR-5, the first CLQA task targeting Chinese, English, and Japanese languages was carried out. Following the success of NTCIR-5 CLQA NTCIR-6 hosted the second campaign on the CLQA task. Since the handling of Named Entities is a major issue in CLQA, we aimed to promote research on cross-lingual Question Answering technology capable of Named Entities in East Asian languages. We conducted evaluations of seven subtasks: E-J, J-J, J-E, E-C, C-C, C-E, and E-E subtask, where C, E, and J stand for Chinese, English, and Japanese, respectively, and X-Y indicates that questions are given in language X and answers are extracted from documents written in language Y. For the purpose of system development, we provided sample question/answer pairs and formal run answer/question pairs used at the previous CLQA task. The Formal Run evaluation was conducted during November 1-7, 2006 with 200 and 150 test questions for Japanese related CLQA and Chinese related CLQA, respectively. As a result, 12 research groups world-wide participated in CLQA, and 91 runs were submitted in total."

• "ICT-DCU Question Answering Task at NTCIR6", Sen Zhang ("IR group, ICT, CAS"), Bin Wang ("IR group, ICT, CAS") and Gareth Jones ("School of Computing,Dublin City University"). (PDF)

"This paper describes details of our participation in the NTCIR-6 Chinese-to-Chinese Question Answering task. We use the Òretrieval plus extraction approachÓ to get answers for questions. We first split the documents into short passages, and then retrieve potentially relevant passages for a question, and finally extract named entity answers from the most relevant passages. For question type identification, we use simple heuristic rules which can cover most questions. The Lemur toolkit was used with the okapi model for document retrieval. Results of our task submission are given and some preliminary conclusions drawn."

• "Two-Pass Named Entity Classification for Cross Language Question Answering", Yu-Chieh Wu (CSIE NCU), Kun-Chang Tsai (CSIE NCU) and Jie-Chi Yang (Graduate Institute of Net-work Learning Technology NCU). (PDF)

"In this paper, we present the mono-lingual and bilingual question answering experi-mental results at NTCIR6-CLQA. We combine most of the online resources and available resources to our QA systems without employing additional resources such as ontology, labeled data. Our method relies on three main important components, namely, passage retrieval, question classifier, and the named entity recognizer. Although our QA model is not state-of-the-art, the attractive of our method is that it was designed fully auto-matic without further adjusting the weights on different keywords. In the bi-lingual retrieval tasks, we translate the queries through a well-known machine translation tool. The evaluation results of our method were also given in the tail of this paper."

• "Chinese-Chinese and English-Chinese Question Answering with ASQA at NTCIR-6 CLQA", Cheng-Wei Lee ("Department of Computer Science, National Tsing-Hua University, Taiwan, R.O.C"), Min-Yuh Day ("Institute of Information Science, Academia Sinica, Taiwan, R.O.C"), Cheng-Lung Sung ("Institute of Information Science, Academia Sinica, Taiwan, R.O.C"), Yi-Hsun Lee ("Institute of Information Science, Academia Sinica, Taiwan, R.O.C"), Tian-Jian Jiang ("Institute of Information Science, Academia Sinica, Taiwan, R.O.C"), Chia-Wei Wu ("Institute of Information Science, Academia Sinica, Taiwan, R.O.C"), Cheng-Wei Shih ("Institute of Information Science, Academia Sinica, Taiwan, R.O.C"), Yu-Ren Chen ("Institute of Information Science, Academia Sinica, Taiwan, R.O.C") and Wen-Lian Hsu ("Institute of Information Science, Academia Sinica, Taiwan, R.O.C"). (PDF) (Slides PDF) (Poster PDF)

"For NTCIR-6 CLQA, we improved our question answering system ASQA (Academia Sinica Question Answering System), which participated in NTCIR-5 CLQA, so that it could deal with the Chinese-Chinese (C-C) subtask and the English-Chinese (E-C) subtask. There are three innovations in the improved system: (a) to handle the E-C subtask, we have built an English question classifier that adopts Question Informer as a key classification feature; (b) with automatically generated Answer Templates, we can accurately pinpoint the correct answers for some questions. When Answer Templates are applied, the RU-accuracy is 0.911 for the applied questions; and (c) the Answer Ranking module has been improved by incorporating a new feature called, SCO-QAT (Sum of Co-occurrence of Question and Answer Terms). In NTCIR-6 CLQA, ASQA achieved 0.553 RU-accuracy in the C-C subtask and 0.34 RU-accuracy in the E-C subtask."

• "A Method of Cross-Lingual Question-Answering Based on Machine Translation and Noun Phrase Translation using Web documents --- Yokohama National University at NTCIR-6 CLQA ---", Tatsunori Mori (Yokohama National University) and Kousuke Takahashi (Yokohama National University). (PDF) (Slides PDF)

"We propose a method of English-Japanese cross lingual question-answering (E-J CLQA) that uses machine translation (MT) and an existing Japanese QA system. We also introduce noun phrase translation using Web documents in order to compensate the insufficiencies in the bilingual dictionary of the MT system. We combine several phrase translation techniques including 1) phrase translation using Wikipedia, 2) phrase translation using Web search results only, 3) phrase translation using Web search results and the information of pronunciation. The experimental result shows that the combination increases the coverage of translation and also improves the accuracy of E-J CLQA. However, the improvement is not so significant because the MT system works well for the NTCIR-6 E-J questions."

• "NTCIR-6 Monolingual Chinese and English-Chinese Cross-Lingual Question Answering Experiments using PIRCS", Kui-Lam Kwok ("Queens College, CUNY"), Peter Deng ("Queens College, CUNY") and Norbert Dinstl ("Queens College, CUNY"). (PDF)

"We continue to employ a minimal approach for our Chinese QA work that requires only a COTS entity extraction software and other home-built tools. In monolingual Chinese QA, questions are classified based on cue-word and meta-keyword usage patterns. Retrieval is done using sentence units, and indexing is based on bigrams and characters. Entities extracted from retrieved sentences form a pool of answer candidates which are ranked using five evidence factors. Our best monolingual result shows that when only Top1 answers are considered, 63 questions out of 150 are answered correctly with sentence support, giving an accuracy and MRR of 0.42. When unsupported answers are included, these values improve to 0.4467. English-Chinese CLQA starts with English question classification also based on an approach similar to Chinese. Three paths of translation render the question into Chinese strings. Otherwise procedures of retrieval and answer ranking remain the same as monolingual but with different parameter values. Our best run returns corresponding Top1 values as: .2533 and .28 (unsupported). These are about 60% of monolingual effectiveness within our system. Effectiveness with Top2-5 answers as well as the influence of different evidence factors are also reported."

• "NTCIR-6 CLQA Question Answering Experiments at the Tokyo Institute of Technology", Josef Novak (Tokyo Institute of Technology), Edward Whittaker (Tokyo Institute of Technology), Matthias Heie (Tokyo Institute of Technology), Shuichiro Imai (Tokyo Institute of Technology) and Sadaoki Furui (Tokyo Institute of Technology). (PDF)

"In this paper we discuss our results from the 2006 NTCIR-6 CLQA task, subtasks 2a and 2b. We describe our language independent, data-driven approach to Japanese language question answering and our new document retrieval and answer projection method which resulted in a small performance gain in comparison to earlier approaches. Using this method, we achieve a formal run score of 0.17 for the top answer with document support for subtask 2b. We achieve a less favorable score of 0.03 for the top answer for the cross language subtask 2a, however we attribute this primarily to deficiencies in third-party MT software utilized for translation. We argue that these results further validate our current approach to QA."

• "JAVELIN III: Cross-Lingual Question Answering from Japanese and Chinese Documents", Teruko Mitamura (Carnegie Mellon University), Frank Lin (Carnegie Mellon University), Hideki Shima (Carnegie Mellon University), Mengqiu Wang (Carnegie Mellon University), Jeongwoo Ko (Carnegie Mellon University), Justin Betteridge (Carnegie Mellon University), Matthew Bilotti (Carnegie Mellon University), Andrew Schlaikjer (Carnegie Mellon University) and Eric Nyberg (Carnegie Mellon University). (PDF)

"In this paper, we describe the JAVELIN Cross Language Question Answering system, which includes modules for question analysis, keyword translation, document retrieval, information extraction and answer generation. In the NTCIR6 CLQA2 evaluation, our system achieved 19% and 13% accuracy in the English-to-Chinese and English-to-Japanese subtasks, respectively. An overall analysis and a detailed module-by-module analysis are presented."

• "Extracting and Ranking Question-Focused Terms Using the Titles of Wikipedia Articles", Yi-Che Chan ("Department of Computer Science and Information Engineering, National Cheng Kung University, Taiwan, R.O.C."), Kuan-Hsi Chen ("Department of Computer Science and Information Engineering, National Cheng Kung University, Taiwan, R.O.C.") and Wen-Hsiang Lu ("Department of Computer Science and Information Engineering, National Cheng Kung University, Taiwan, R.O.C."). (PDF)

"At the NTCIR-6 CLQA (Cross-Language Question Answering) task, we participated in the Chinese-Chinese (C-C) and English-Chinese (E-C) QA (Question Answering) subtasks. Without employing question type classification, we proposed a new resource, Wikipedia, to assist in extracting and ranking Question-Focused terms. We regarded the titles of Wikipedia articles as a multilingual noun-phrase corpus which is useful in QA systems. Experimental results showed that better performance was achieved for questions with type PERSON or LOCATION. Besides, we used an online MT (Machine Translation) system to deal with question translation in our CLQA task."

• "Statistical Machine Translation based Passage Retrieval for Cross-Lingual Question Answering --- Experiments at NTCIR-6", Tomoyosi Akiba (Toyohashi University of Technology), Kei Shimizu (Toyohashi University of Technology), Atsushi Fujii (University of Tsukuba) and Katunobu Itou (Hosei University). (PDF)

"In this paper, we propose a novel approach for Cross-Lingual Question Answering (CLQA), where the statistical machine translation (SMT) is utilized. In the proposed method, the SMT is deeply incorporated into t he question answering process, instead of using it as the pre-processing of the mono-lingual QA process as in the previous work. The proposed method can be considered as exploiting the SMT-based passage retrieval for CLQA task. Our experimental results targeting the English-to-Japanese CLQA using the NTCIR CLQA 1 and 2 test collections showed that the proposed method outperformed the previous pre-translation approach."

• "Utilizing Entity Relation to Bridge the Language Gap in Cross-Lingual Question Answering System", Min Wu ("Institute of Informatics, Logics and Security Studies, University at Albany, SUNY") and Tomek Strzalkowski ("Institute of Informatics, Logics and Security Studies, University at Albany, SUNY"). (PDF)

"We describe University at Albany's CLQA system and its performance in English-Chinese subtask evaluation in NTCIR-6 CLQA. Firstly we illustrate our submitted system, which was built in two weeks. (We had to finish our CLQA system in this time limit because we were late registered.) Then we would like to introduce the improved system which utilizes our ACE (Automatic Content Extraction) relation detection and recognition (RDR) system to help bridge the language gap when answering some types of questions. The experiment result shows that our proposed method helps a lot to improve the system performance in answering questions that seek some specific relation between entities."

• "Heuristic and Syntactic Scoring for Cross-language Question Answering", Lisa Ballesteros (Mount Holyoke College) and Xiaoyan Li (Mount Holyoke College). (PDF)

"This paper describes the Marsha Cross-language Question Answering System used by Mount Holyoke College in the English-Chinese, Chinese-Chinese, and English-English subtasks of the NTCIR Cross-lingual Question Answering task. The system was most effective in the Chinese and English monolingual tasks. However, improved translations and better query type identification remain challenges for more effective cross-language QA task performance."

• "Expansion of Multimodal Summarization for Trend Information -- Report on the first and second cycles of the MuST Workshop --", Tsuneaki Kato (University of Tokyo), Mitsunori Matsushita (NTT Communications Science Laboratories, NTT Corp.) and Noriko Kando (National Institute of Informatics). (PDF)

A trend, which is a general tendency in the way a situation is changing or developing, is based on temporal statistical data, and can be obtained by synthetically summarizing it. A system that can properly answer questions on interesting trends must help users to understand and review a huge amount of information. MuST, a workshop on multimodal summarization for trend information, was designed to encourage co-operative and competitive studies on multimodal summarization for trend information, and to allow it to construct such an information system. MuST is a pilot workshop of the NTCIR workshop. It has been run based on approximately one year as one cycle, and it finished its second cycle in March 2007. Several researches with broader scope than initially expected have been conducted in the workshop. This paper explains the framework and objective of the MuST workshop, reviews researches conducted there and the results obtained so far, and discusses the role of the workshop for those researches.

• "Extraction and Visualization of Trend Information from Newspaper Articles and Blogs", Hidetsugu Nanba (Hiroshima City University), Nao Okuda (Hiroshima Elpida Memory, Inc.) and Manabu Okumura (Tokyo Institute of Technology). (PDF)

Trend information is a summarization of temporal statistical data, such as changes in product prices and sales. We propose a method for extracting trend information from multiple newspaper articles and blogs, and visualizing the information as graphs. As target texts for extraction of trend information, the MuST (Multimodal Summarization for Trend Information) workshop focuses on newspaper articles. In addition to newspapers, we focus on blogs, because useful information for analysing trend information is often written in blogs, such as the reasons for increases/decreases of statistics and the impact of increases/decreases of statistics on society. To extract trend information, we extract temporal expressions and statistical values, and we devised methods for both operations. To investigate the effectiveness of our methods, we conducted some experiments. We obtained a recall of 6.3% and precision of 31.3% for newspaper articles, and a recall of 44.8% and precision of 60.3% for blogs. From the error analysis, we found that most errors in newspaper articles were caused by misconversion of temporal expressions such as “同年” (the same year) or “前月” (the previous month), into “YYYY-MM-DD” form, although temporal expressions were detected correctly. In contrast to newspaper articles, there are few temporal expressions in blogs for which resolution is required, such as “同日” (the same day) or “前月” (the previous month). As a result, recall and precision for blogs are higher than those for newspaper articles.

• "Visualization of Earthquake Trend Information from MuST Corpus", Yasufumi Takama (Tokyo Metropolitan University). (PDF)

A system for extracting and visualizing trend information about earthquakes from tagged corpus, which is distributed by a Workshop on Multimodal Summarization for Trend Information (MuST), is proposed. The topic of earthquakes does not contain only temporal trends, which are main concerns of typical topics such as gas price and stock price movement, but also spatial trends including the seismicity of earthquakes. The proposed system employs the map of Japan for visualizing spatial trends, as well as bar and line charts for temporal trends. The system also focuses on swarm earthquakes, which are visualized with the combination of the map and bar chart visualizations. Furthermore, the system provides users with an interactive facility so that they can obtain several visualization results with intuitive operations such as mouse-click on the map. A prototype system is implemented, of which functionality is compared with existing earthquake database system. The merit of extracting earthquake trend inforfrom newspaper articles is also considered.

• "Extraction of Statistical Terms and Co-occurrence Networks from Newspapers", Haruka Saito ("Service Platforms Res. Lab., NEC"), Hideki Kawai ("C&C Innovation Res. Lab., NEC"), Masaaki Tsuchida ("Service Platforms Res. Lab., NEC"), Hironori Mizuguchi ("Service Platforms Res. Lab., NEC") and Dai Kusui ("Service Platforms Res. Lab., NEC"). (PDF)

In this paper, we automatically extract statistical terms and build their co-occurrence networks from newspapers. Statistical terms are expression of the measurements of statistics to watch the movements of phenomena; birth rates, public approval rating of the Cabinet and so on. In recent years, we have a vast amount of available information because of computerization and the technologies of making their overview and enhancement of their values are noticed. One of them is the technology of visualizing information of social trend and movements from newspapers. For visualizing trend information, there are two approaches. One is a visualization of the values of statistical terms based on information on trends. For example, this approach draws graphs of the temporal movements of statistics or the geographical distribution of statistics. The other is a visualization of relations among the statistical terms, for example, extracting a causal relation among statistical terms and showing with networks. In this paper, we take the latter approach of building networks of causal relations among the statistical terms. To extract statistical terms, we propose extraction method using suffixes. To extract causal relations among statistical terms, we first extract co-occurrence relations and next show them with the networks. We can extract many statistical terms with high accuracy by our method and find interesting links among some statistical terms by our co-occurrence networks.

• "Extraction of important numerical pairs from text documents and visualization of them", Masaki Murata (NICT), Ko ji Ichii (Hiroshima Univ.), Qing Ma (Ryukoku Univ.), Tamotsu Shirado, Toshiyuki Kanamaru, Sachiyo Tsukawaki and Hitoshi Isahara (NICT). (Poster PDF)

• "Overview of Opinion Analysis Pilot Task at NTCIR-6", Yohei Seki (Toyohashi University of Technology), David Kirk Evans (National Institute of Informatics), Lun-Wei Ku (National Taiwan University), Hsin-Hsi Chen (National Taiwan University), Noriko Kando (National Institute of Informatics) and Chin-Yew Lin (Microsoft Research Asia). (Paper PDF) (Slides PDF)

This paper describes an overview of the Opinion Analysis Pilot Task from 2006 to 2007 at the Sixth NTCIR Workshop. We created test collection for 32, 30, and 28 topics (11,907, 15,279, and 8,379 sentences) in Chinese, Japanese and English. Using this test collection, we conducted opinion extraction subtask. The subtask was defined from four perspectives: (a) opinionated sentence judgment, (b) opinion holder extraction, (c) relevance sentence judgment, and (d) polarity judgment. 21 run results were submitted by 14 participants with five results submitted by the organizers. We show the evaluation results of the groups participating in opinion extraction subtask.

• "Appraisal Extraction for News Opinion Analysis at NTCIR-6", Kenneth Bloom (Illinois Institute of Technology), Sterling Stein (Illinois Institute of Technology) and Shlomo Argamon (Illinois Institute of Technology). (PDF)

"We describe a system which uses lexical shallow parsing to find adjectival ``appraisal groups'' in sentences, which convey a positive or negative appraisal of an item. We used a simple heuristic to detect opinion holders, determining whether a person was being quoted in a specific sentence or not, and if so, who. We also explored the the use of unsupervised learners and voting to increase our coverage."

• "Cornell System Description for the NTCIR-6 Opinion Task", Eric Breck (Cornell University), Yejin Choi (Cornell University), Veselin Stoyanov (Cornell University) and Claire Cardie (Cornell University). (PDF)

"We present our opinion analysis system for English that was used in the Opinion Analysis Pilot Task at NTCIR-6. Our goal in developing the system was to use, as much as possible, components and features from our previous work in this area."

• "A low-resources approach to Opinion Analysis: Machine Learning and Simple Approaches", David Kirk Evans (National Institute of Informatics). (PDF)

"In this paper I present a system for automatic opinion analysis built in a short time-frame using freely available open-source processing tools and lexical resources available from prior research. I use a simple feature-set that is largely language independent and a freely available machine-learning framework to model the subtasks as classification problems and report on my system's performance. Additionally, I show that blind relevance feedback improves results sentence-level relevance judgment. My system shows that it is possible to quickly build an opinion analysis system in a short period of time that can perform at an average level."

• "ISCAS in Opinion Analysis Pilot Task: Experiments with sentimental dictionary based classifier and CRF model", Rui-hong Huang (ISCAS), Le Sun (ISCAS) and Long-xi Pan (ISCAS). (PDF)

"The paper presents our work in the opinion pilot task in NTCIR6 in Chinese. In extracting opinion holders, we applied Conditional Random Field (CRF) model to find the opinion holders as a sequential labeling task, while in determining the subjectivity and the polarity, we adopted a simple empirical algorithms based on the sentimental dictionary to discriminate the subjective sentences from the objective ones and suggest their polarities. Besides the features used in the CRF model and the detailed specification in the machine learning system, the evaluation results and the error analysis will also be presented."

• "Japanese Opinion Extraction System for Japanese Newspapers Using Machine -Learning Method", Toshiyuki Kanamaru (National Institute of Information and Communications Technology), Masaki Murata (National Institute of Information and Communications Technology) and Hitoshi Isahara (National Institute of Information and Communications Technology). (PDF)

"We constructed a Japanese opinion extraction system for Japanese newspaper articles using a machine-learning method for the system. We used opinion-annotated articles as learning data for the machine- learning method. The system extracts opinionated sentences from newspaper articles, and specifies opinion holders and opinion polarities of the extracted sentences. The system also evaluates whether or not the sentences of the articles are relevant to the given topic. We conducted experiments using the NTCIR-6 opinion extraction subtask data collection and obtained the following accuracy rates using a lenient gold standard: opinion extraction, 42.88%; opinion holder extraction, 14.31%; polarity decision, 19.90%; and relevance evaluation, 63.15%."

• "Opinion Analysis based on Lexical Clues and their Expansion", Youngho Kim (Information and Communications University) and Sung-Hyon Myaeng (Information and Communications University). (PDF)

"The challenge of an automatic opinion analysis has been the focus of attention in recent years in many domains such as online product review. Especially, in online news articles opinion analysis has good prospects, since newspaper is the most powerful media to disseminate people's opinions. We introduce a lexical information based approach to this task by exploiting lexical information, based on the quantitative analysis of opinions in the news articles. The method comprises semi-supervised subjectivity classification, gloss based sentiment classification, and rule based opinion holder finder. The method we present is remarkable since numbers of lexical clues we discovered were effective to this task. The experimental results show that our system achieves 45% of performance to extract opinionated sentences and 35% of performance to identify opinion holders."

• "Using Opinion Scores of Words for Sentence-Level Opinion Extraction", Lun-Wei Ku (National Taiwan University), Yong-Sheng Lo (National Taiwan University) and Hsin-Hsi Chen (National Taiwan University). (PDF)

"The opinion analysis task is a pilot study task in NTCIR-6. It contains the challenges of opinion sentence extraction, opinion polarity judgment, opinion holder extraction and relevance sentence extraction. The three former are new tasks, and the latter is proven to be tough in TREC. In this paper, we introduce our system for analyzing opinionated information in NTCIR-6 document collections. Several formulae are proposed to decide the opinion polarities and strengths of words from composed characters and then further to process opinion sentences. The negation operators are also taken into consideration in opinion polarity judgment, and the opinion operators are used as clues to find the locations of opinion holders. The performance of the opinion extraction and polarity judgment achieves the f-measure 0.383 under the lenient metric and 0.180 under the strict metric, which is the second best of all participants."

• "Experiments of Opinion Analysis on the Corpora MPQA and NTCIR-6", Yaoyong Li ("Department of Computer Science, The University of Sheffield"), Kalina Bontcheva ("Department of Computer Science, The University of Sheffield") and Hamish Cunningham ("Department of Computer Science, The University of Sheffield"). (PDF)

"This paper describes the algorithms and linguistic features used in our participating system for the opinion analysis pilot task at NTCIR-6. It presents and discusses the results of our system on the opinion analysis task. It also presents our experiments of opinion analysis on the two corpora MPQA and NTCIR-6, by using our learning based system. Our system was base on the SVM learning. It achieved state of the art results on the MPQA corpus for the two problems, opinionated sentence recognition and opinion holder extraction. The results using the NTCIR-6 English corpus for both training and testing are certainly among the first ones. Our results on the opinionated sentence recognition sub-task of the NTCIR-6 were encouraging. The results on the English evaluation of the NTCIR-6 opinion analysis task were obtained from the models learned from the MPQA corpus. The lower results on the NTCIR-6 opinion holder extraction sub-task, in comparison with those using each corpus for both training and testing, may possibly show that there exist substantial differences between the MPQA corpus and the NTCIR-6 English corpus."

• "Three-Phase Opinion Analysis System at NTCIR-6", Hironori Mizuguchi (NEC Internet Systems Research Laboratory), Masaaki Tsuchida (NEC Internet Systems Research Laboratory) and Dai Kusui (NEC Internet Systems Research Laboratory). (PDF)

"We developed an opinion analysis system at NTCIR-6. Our system can detect opinion sentences and extract opinion holders by executing three phases: (1) opinion sentence classification by SVM that distinguishes an opinionated sentence from others, (2) opinion-holder candidate extraction using named entity recognition, (3) opinion-holder detection by rules that find the correspondence between the sentence and the holder. Characteristics of the system are the following two points: (a) in phase 1, the end of a sentence expression is added to the feature in SVM vector, and (b) phase 3 is separated into the author detection and the others. As a result of the evaluation, in opinion sentence judgment both precision and the recall ratio improved based on point (a). In opinion holder extraction, precision has improved greatly based on point (b)."

• "Crosslingual Opinion Extraction from Author and Authority Viewpoints at NTCIR-6", Yohei Seki (Toyohashi University of Technology). (PDF)

"Opinion research has been paid much attention by the Information Retrieval (IR) community, and opinion holder extraction research is important for discriminating between opinions that are viewed from different perspectives. In this paper, we describe our experience of participation in the {\it NTCIR-6 Opinion Analysis Pilot Task} by focusing on opinion extraction results in Japanese and English. Our approach to opinion holder extraction was based on the discrimination between author and authority viewpoints in opinionated sentences, and the evaluation results were fair with respect to the Japanese documents."

• "NTCIR-6 at Maryland: Chinese Opinion Analysis Pilot Task", Yejun Wu ("College of Information Studies and Institute for Advanced Computer Studies, University of Maryland, College Park") and Douglas Oard ("College of Information Studies and Institute for Advanced Computer Studies, University of Maryland, College Park"). (PDF)

"For the Chinese opinion analysis pilot task at NTCIR-6, we tested two techniques for each of the four subtasks---identifying opinionated sentences, making polarity decisions, identifying opinion holders, and retrieving topically relevant sentences. Our opinion detection technique is based on sentiment lexicons. We explored three main issues: the effect of the size of sentiment lexicons on the accuracy of opinionated sentence identification and of polarity decisions, the effect of a simple approximation to anaphora resolution on the accuracy of opinion holder identification, and the effect of sentence expansion on the effectiveness of relevant sentence retrieval."

• "Opinmine - Opinion Analysis System by CUHK for NTCIR-6 Pilot Task", Ruifeng Xu ("Department of S.E.E.M., The Chinese University of Hong Kong"), Kam-Fai Wong ("Department of S.E.E.M., The Chinese University of Hong Kong") and Yunqing Xia ("Department of S.E.E.M., The Chinese University of Hong Kong"). (PDF)

"This paper presents the CUHK opinion analysis system, namely Opinmine, for the NTCIR-6 pilot task. Opinmine comprises of three functional modules: (1) Preprocessing and Assignment Module (PAM) performs word segmentation, part-of-speech (POS) tagging and named entity recognition on the input Chinese text. It is based on lexicalized Hidden Markov Model and heuristic rules. (2) Knowledge Acquisition Module (KAM) applies unsupervised learning techniques to acquire different opinion knowledge including opinion operator, opinion indicator and opinion words from annotated data and Web data. (3) Sentence Analysis Module (SAM) analyzes each input sentence to determine whether it is opinionated. For each opinionated sentence, its opinion holders, opinion operators and opinion words are recognized and its polarity is determined. Furthermore, the relevance between the sentence and a topic are judged by based on sentence-topic and document-topic relevance. For lenient evaluation, the F1 performance of Opinmine in opinion extraction, polarity decision and relevance judgment are 0.635, 0.405 and 0.812, respectively; and for strict evaluation, the F1 performances are 0.427, 0.296 and 0.616, respectively."

• "Overview of the Patent Retrieval Task at the NTCIR-6 Workshop", Atsushi Fujii (University of Tsukuba), Makoto Iwayama (Hitachi, Ltd./Tokyo Institute of Technology) and Noriko Kando (National Institute of Informatics). (PDF)

"In the Sixth NTCIR Workshop, we organized the Patent Retrieval Task and performed three subtasks; Japanese Retrieval, English Retrieval, and Classification. This paper describes the Japanese Retrieval Subtask and English Retrieval Subtask, both of which were intended for patent-to-patent invalidity search task. We show the evaluation results of the groups participating in those subtasks."

• "Overview of Classification Subtask at NTCIR-6 Patent Retrieval Task", Makoto Iwayama ("Hitachi, Ltd./Tokyo Institute of Technology"), Atsushi Fujii (University of Tsukuba) and Noriko Kando (National Institute of Informatics). (PDF)

"This paper describes the Classification Subtask of the NTCIR-5 Patent Retrieval Task. The purpose of this subtask is to evaluate the methods of classifying patents into multi-dimensional classification structures called F-term (File Forming Term) classification systems. We report on how this subtask was designed, the test collection released, and the results of the evaluation."

• "Leveraging Category-based LSI for Patent Retrieval", Masaki Aono (Toyohashi University of Technology). (PDF)

"Latent Semantic Indexing (LSI) has been employed to reduce dimension of indices of documents for similarity search. In this paper, we will describe a method for retrieving conceptually similar patents first by categorizing patent collection and then by applying LSI algorithm multiple times to each category. The main strategy is keeping the algorithm as simple as possible, while achieving the scalability for massive dataset. During the categorization phase, we allow any patent to be classified into multiple categories, which allows patent document overlaps among different categories. Then, for each category, we applied dimensional reduction using LSI to each category into a much lower dimension. Finally, once a query as a collection of claim sentences for a patent is given, we select the most similar category, and return top fifty ranked patent documents as candidates to invalidate the query document."

• "Integrating Content and Citation Information for the NTCIR-6 Patent Retrieval Task", Atsushi Fujii (University of Tsukuba). (PDF)

"This paper describes our system participated in the Japanese and English Retrieval Subtasks at the NTCIR-6 Patent Retrieval Task. The purpose of these subtasks is the invalidity search, in which a patent application including a target claim is used to search documents that can invalidate the demand in the claim. Although we use a regular text-based retrieval method for the Japanese Retrieval Subtask, we combine text and citation information to improve the retrieval accuracy for the English Retrieval Subtask."

• "Multi-label Patent Classification at NTT Communication Science Laboratories", Akinori Fujino (NTT Communication Science Laboratories) and Hideki Isozaki (NTT Communication Science Laboratories). (PDF)

"We design a multi-label classification system based on the combination of binary classifications for classification subtask at NTCIR-6 Patent Retrieval Task. In our system, we design a binary classifier per F-term that determines the assignment of the F-term to patent documents. Hybrid classifiers are employed as binary classifiers so that the multiple components of patent documents are used effectively. The hybrid classifiers are constructed by combining component generative models with weights based on the maximum entropy principle. Using a test collection of Japanese patent documents, we confirmed that our system provided good ranking of F-terms as regards assigning them to patent documents."

• "Term Weighting Classification System Using the Chi-square Statistic for the Classification Subtask at NTCIR-6 Patent Retrieval Task", Kotaro Hashimoto (University of Technology) and Takashi Yukawa (University of Technology). (PDF)

"In the present paper, a term weighting classification method using the chi-square statistic is proposed and evaluated in the classification subtask at NTICR-6. In this task, large numbers of patent applications are classified into F-term categories. Therefore, a patent classification system requires high classification speed, as well as high classification accuracy. The proposed method is evaluated in A-precision, R-precision, and F-measure. Although the proposed method did not obtain the best score, this method achieves a classification accuracy that is as high as those of other methods using machine learning or the vector classification method. In this task, the processing speed is not evaluated. Therefore, processing speed is also evaluated. The evaluation results show that the proposed method is much faster than that using the vector classification method. Evaluation results of classification accuracy and processing speed show that the proposed method is confirmed to be effective and to be practical."

• "Invalidity Search for USPTO Patent Documents Using Different Patent Surrogates", Yuen-Hsien Tseng (National Taiwan Normal University), Chen-Yang Tsai (Fu Jen Catholic University) and Da-Wei Juang (WebGenie Information LTD.). (PDF)

"This paper describes our work at the sixth NTCIR workshop on the subtask of invalidity search for patent retrieval. We compared different patent surrogates for their effectiveness on invalidity search. Our preliminary results show that the query by the Claims field plus PRF (pseudo relevance feedback) leads to the best results in terms of relevance degree A while the query by all free-text fields yields highest performance under relevance degree B."

• "POSTECH at NTCIR-6 English Patent Retrieval Subtask", Jungi Kim (Pohang University of Science and Technology), Ye-Ha Lee (Pohang University of Science and Technology), Seung-Hoon Na (Pohang University of Science and Technology) and Jong-Hyeok Lee (Pohang University of Science and Technology). (PDF)

"This paper reports our experimental results at the NTCIR-6 English Patent Retrieval Subtask. Our previous participation at the patent retrieval subtask revealed that the long length of the patent applications require less smoothing of the document model than general documents such as news paper articles. We setup the initial baseline retrieval system for U.S. patent applications and compare the difference from that of Japanese patent applications"

• "SVM Based Learning System for F-term Patent Classification", Yaoyong Li (The University of Sheffield), Kalina Bontcheva (The University of Sheffield) and Hamish Cunningham (The University of Sheffield). (PDF)

"This paper describes our SVM-based system and the techniques we used to adapt the approach for the specifics of the the F-term patent classification subtask at NTCIR-6 Patent Retrieval Task. Our system obtained the best results according to two of the three measures used for performance evaluation. Moreover, the results from some additional experiments demonstrate that our system has benefited from the SVM adaptations which we carried out. It also benefited from using the full patent text in addition to using the F-term description as extra training material. However, our results using an SVM variant designed for hierarchical classification were much worse than those achieved with flat SVM classification. At the end of the paper we discuss the possible reasons for this, in the context of the F-term classification task."

• "NTCIR-6 Patent Retrieval Experiments at Hitachi", Hisao Mase ("Hitachi, Ltd.") and Makoto Iwayama ("Hitachi, Ltd."). (PDF)

"Our goal in NTCIR-6 was to find the accuracy limitations of our current patent retrieval techniques. Thus, we focused only on optional runs, in which not only the claim text used in mandatory runs, but also other texts in a query patent can be used as the input data. We applied six retrieval methods step by step: (1) TF-IDF-based term weighting using TF in a whole query patent text, (2) adding terms extracted from an abstract to the query terms, (3) adding terms with higher weights extracted from a whole patent text, (4) term weight tuning based on the term co-occurrence, (5) document filtering using theme codes, one of the patent classifications, and (6) similarity score tuning using theme codes. Although these methods are simple, we found they are powerful enough to dramatically improve both recall and precision in comparison with a baseline method. By combining above methods, the MAPs improved by a maximum of 47% in comparison to the baseline method."

• "Using the K-Nearest Neighbor Method and SMART Weighting in the Patent Document Categorization Subtask at NTCIR-6", Masaki Murata (National Institute of Information and Communications Technology), Toshiyuki Kanamaru (National Institute of Information and Communications Technology), Tamotsu Shirado (National Institute of Information and Communications Technology) and Hitoshi Isahara (National Institute of Information and Communications Technology). (PDF)

"Patent processing is extremely important in industry, business, and law. We participated in an F-term categorization subtask at NTCIR-6, in which, we classified patent documents into their F-terms using the k-nearest neighbor method. For document classification, F-term categories are both very precise and useful. We entered five systems in the F-term categorization subtask and obtained good results with them. Thus, we confirmed the effectiveness of our method. By comparing various similarity calculation methods, we confirmed that the SMART weighting scheme was the most effective method in our experiments."

• "Query Expansion using an Automatically Constructed Thesaurus", Hidetsugu Nanba (Hiroshima City University). (PDF)

"Our group participated in the Japanese and Eng-lish Retrieval Subtasks of NTCIR-6. Our goal was to evaluate the effectiveness of a thesaurus constructed from patents for invalidity search. To confirm the effectiveness of our thesaurus-based query expansion, we conducted experiments and found that our method can improve upon traditional document retrieval systems."

• "F-term classification Experiments at NTCIR-6 for Justsytems", Masaki Rikitoku (Justsystems). (PDF)

"We conducted the classification subtask at NTCIR-6 Patent Retrieval Task using a system based on three document classifiers, namely, a one-vs-rest SVM classifier, multi-topic classifier, and binary Naive Bayes classifier. The multi-topic classifier was constructed on the basis of the maximum margin principle and applied to multiple F-term classification. From the experimental results, this multi-topic classifier yielded a higher F1 value than the one-vs-rest SVM in many cases. In addition, we employed the one-vs-rest SVM classifier. The SVM classifier has certain drawbacks such as low recall performance and large learning time. In order to solve these problems, we used heuristics for achieving random reduction of a part of the negative examples and division of learning. These procedures lead to a reduction in learning time and improve the classification performance when appropriate parameters are set."

• "A Passage Retrieval System using Query Expansion and Emphasis", Hiroki Tanioka (JustSystems Corporation) and Kenichi Yamamoto (JustSystems Corporation). (PDF)

"We developed a patent retrieval system with the corresponding very large number of patents from NTCIR-6 Patent Retrieval Task. And we developed a method of refining and emphasizing query. Our retrieval system consisting of four PCs could make indices of all claims in specifications for ten years. Then we confirmed that the query emphasis was better mean average precision than merely query expansion. And we had tried to reduce the number of results with the belief assessment."

• "F-term Classification System Using K-Nearest Neighbor Method", Kazuya Konishi (NTT Data Corporation) and Toru Takaki (NTT Data Corporation). (PDF)

In the Classification subtask of the NTCIR-6 Patent Retrieval Task, we implemented an F-term classification system using the k-nearest neighbor method. This system is based on the hypothesis that an F-term assigned to many patent documents that are similar to the topic patent document should also be assigned to the topic patent document. In implementing this system, we considered and applied methods for calculating similarity between patent documents, extracting patent documents from the training set, and ranking F-terms to the F-term classification system. In this paper, we report the result of F-term classification.

• "An Overview of the 4th Question Answering Challenge (QAC-4) at NTCIR Workshop 6", Junichi Fukumoto (Ritsumeikan University), Tsuneaki Kato (University of Tokyo), Fumito Masui (Mie University) and Tatsunori Mori (Yokohama National University). (Official Proceedings Paper PDF) (Revised Paper 2007-05-21) (Slides PDF)

"In QAC-4, we defined question answering task using any type of question, mainly focused on non-factoid questions. There are 8 participants and 14 runs from these participants. In the evaluation, four kinds of criterion were used for some portion of participants answer set. The evaluation results showed some of the participant systems could focus on the area which correct answer contents exist but have tendency to fail to extract correct answer areas. It is caused by complex question types and difficulty of correct answer scope extraction."

• "Question Answering System for Non-factoid Type Questions and Automatic Evaluation based on BE Method", Junichi Fukumoto (Ritsumeikan University). (Official Proceedings PDF) (Revised Paper 2007-05-21) (Slides PDF)

"In this paper, we describe answer extraction method for non-factoid questions. We classified non-factoid type questions into three types: why type, definition type and how type. We analyzed each type of questions and developed answer extraction patterns for these types of questions. For automatic evaluation, we have developed BE based evaluation tool for answers of questions. BE method is originally proposed by Hovy et. and we applied BE method for question answering evaluation. Evaluation is done by comparison between BEs of system answer and BEs of correct answers."

• "QA System Metis Based on Semantic Graph Matching at NTCIR 6", Minoru Harada (Aoyama Gakuin University), Yuhei Kato (Aoyama Gakuin University), Kazuaki Takehara (Aoyama Gakuin University), Masatsuna Kawamata (Aoyama Gakuin University), Kazunori Sugimura (Aoyama Gakuin University) and Junichi Kawaguchi (Aoyama Gakuin University). (PDF)

"We have developed Metis, a question-answering system that finds an answer by matching a question graph with the knowledge graphs. The question graph is obtained as a result of semantic analysis of a question sentence, the knowledge graphs are similarly analyzed from knowledge sentences retrieved from a database using keywords extracted from the question sentence. In retrieving such knowledge sentences, the system searches for and collects them using Lucene, a search engine, based on search keywords extracted from the question graph. To extract the answer, Metis calculates the degrees of similarity between the question and knowledge graphs to conduct precise matching. In this matching, the system calculates the degrees of similarity, which is the relative size of the similarity co-occurrence graph to the question graphs with respect to all combinations of nodes in the knowledge graph corresponding to those in the question graph. The system then chooses the knowledge graph with the highest degree of similarity and extracts from it the portion that corresponds to the given interrogative word. The system presents this portion as the answer."

• "NTT's Question Answering System for NTCIR-6 QAC-4", Ryuichiro Higashinaka (NTT Communication Science Laboratories) and Hideki Isozaki (NTT Communication Science Laboratories). (PDF)

"NTCIR-6 QAC-4 organizers announced that there would be no restriction (such as factoid) on QAC4 questions, but they plan to include many `definition' questions and `why' questions. Therefore, we focused on these two question types. For `definition' questions, we used a simple pattern-based approach. For `why' questions, hand-crafted rules were used in previous work for answer candidate extraction. However, such rules greatly depend on developers' intuition and are costly to make. We adopt a supervised machine learning approach. We collected causal expressions from the EDR corpus and trained a causal expression classifier, integrating lexical, syntactic, and semantic features. The experimental results show that our system is effective for `why' and `definition' questions."

• "JAVELIN III: Answering Non-Factoid Questions in Japanese", Hideki Shima (Carnegie Mellon University) and Teruko Mitamura (Carnegie Mellon University). (PDF)

"In this paper, we describe our adaptation of the JAVELIN system to Japanese question answering for the NTCIR-6 QAC track. To establish a baseline Japanese-to-Japanese non-factoid question answering system, we performed the minimum extensions to our factoid question answering system. The answer boundary recognition task was simplified by introducing a Òone sentence assumptionÓ in the answer extraction phase. In the end, the performance of our machine learning-based sub-modules was affected by a scarcity of training data; nevertheless, our system performed close to the average accuracy in the number of questions correctly answered."

• "A Monolithic Approach and a Type-by-Type Approach for Non-Factoid Question-answering --- Yokohama National University at NTCIR-6 QAC ---", Tatsunori Mori (Yokohama National University), Mitsuru Sato (Yokohama National University), Madoka Ishioroshi (Yokohama National University), Yugo Nishikawa (Yokohama National University), Shigenori Nakano (Yokohama National University) and Kei Kimura (Yokohama National University). (PDF) (Slides PDF)

"In order to process non-factoid questions in NTCIR-6 QAC, we introduced two types of approaches. First one has a monolithic architecture that retrieves answer passages related to a question using lexical chain. The other one has a type-by-type architecture and consists of four subsystems: i) the subsystem for definitional and other-type questions, ii) the subsystem for why-type questions, iii) the subsystem for how-type questions, and iv) the subsystem for factoid questions. Each of two approaches is based on a common hypothesis that the appropriateness of answer candidate for non-factoid questions can be measured by the combination of a) appropriateness of writing style and b) relevance to the question."

• "A System for Answering Non-Factoid Japanese Questions by Using Passage Retrieval Weighted Based on Type of Answer", Masaki Murata (National Institute of Information and Communications Technology), Sachiyo Tsukawaki (National Institute of Information and Communications Technology), Toshiyuki Kanamaru (National Institute of Information and Communications Technology), Qing Ma ("National Institute of Information and Communications Technology, Ryukoku University") and Hitoshi Isahara (National Institute of Information and Communications Technology). (PDF)

"We constructed a system for answering non-factoid Japanese questions. We used passage retrieval methods for the system. We extracted paragraphs based on terms from an input question and output them as the desired answer. We classified the non-factoid questions into six categories. We used a particular method for each category. For example, we increased the scores of paragraphs including the word "reason" for questions including the word "why." We performed experiments using the NTCIR-6 QAC-4 data collection and tested the effectiveness of our methods."

• "How We Did How, What and Why - HOMIO's Participation in QAC4 of NTCIR-6", Yasutomo Kimura (Otaru University of Commerce), Kenji Ishida (Mie University), Hirotaka Imaoka (Mie University), Fumito Masui (Mie University), Keisuke Kameyama (Hokkaido University), Rafal Rzepka (Hokkaido University) and Kenji Araki (Hokkaido University). (PDF)

"In our paper we describe our second collective challenge to NTCIR-6 Question Answering Challenge (QAC4). Also this time we decided to investigate the limits of the "as automatic as possible" approach to QA. Three teams of Otaru University of Commerce, Mie University and Hokkaido University concentrated on three new question types and the last team also remodeled its WWW Verifier to cope with these types. We will introduce our ideas and methods and then conclude with results and a proposal of further innovations."

• "Non-factoid Question Answering Experiments at NTCIR-6: Towards Answer Type Detection for Realworld Questions", Junta Mizuno (Toyohashi University of Technology), Tomoyosi Akiba (Toyohashi University of Technology), Atsushi Fujii (University of Tsukuba) and Katunobu Itou (Hosei University). (PDF)

"In this paper, we investigate the answer type detection methods for realizing the Universal Question Answering (UQA), which returns an answer for any given question. For this purpose, the questions collected from a WWW question portal community site were analyzed to see how many kinds of questions were submitted in the real world. Then, we introduce the approach for UQA and proposed two methods for the answer type detection. The experimental evaluation using the NTCIR QAC4 test collection showed that the method using one binary classifier that detected the consistency between the types of the given question and the answer candidates was effective."

This section contains supplemental material for the tasks such as additional graphs, evaluation data results, scoring programs, and so on. While some types of data (lists of questions or topics) might be listed, the corpus data sets and test run data are not included. Please check the main NTCIR homepage for information about licesing past corpora and result sets.

• CLIR (English), CLIR (日本語)
• CLQA
• Opinion (English), Opinion (日本語)
• Patent
• QAC