Efficient analysis of patent literature is crucial for technological development and protecting intellectual property. Akey task is verifying the “support requirement,” which mandates that the detailed description must fully describe theclaimed invention. This requirement is fundamental to a patent’s validity. Manual verification is a labor-intensiveprocess that demands technical and legal expertise, making automation highly desirable. However, research onthis task has been hampered by two key challenges: (1) the absence of a public benchmark, and (2) the reliance ofprior work on lexical matching, which fails to capture semantic equivalence. To address these issues, we introduceJAPAS, the first public benchmark for this task, comprising over 2,000 instances manually annotated for Japanesepatents. Each instance is labeled with a claim span, a supporting description paragraph, a relation type, and theannotator’s confidence level. Using this benchmark, we also establish modern baselines that capture semanticsimilarity, such as embeddings and LLMs. Our experiments show that a fine-tuned Qwen3-14B model achieves anF1 score of 0.50, outperforming the conventional lexical-based baseline. This result, which demonstrates that thetask is feasible yet challenging, highlights the utility of JAPAS as a research foundation and provides a performancetarget for future work.

Strongly human-correlated evaluation metrics serve as an essential compass for the development and improvement of generation models and must be highly reliable and robust. Recent embedding-based neural text evaluation metrics, such as COMET for translation tasks, are widely used in both research and development fields. However, there is no guarantee that they yield reliable evaluation results due to the black-box nature of neural networks. To raise concerns about the reliability and safety of such metrics, we propose a method for finding a single adversarial text in the discrete space that is consistently evaluated as high-quality, regardless of the test cases, to identify the vulnerabilities in evaluation metrics. The single hub text found with our method achieved 79.1 COMET% and 67.8 COMET% in the WMT'24 English-to-Japanese (En–Ja) and English-to-German (En–De) translation tasks, respectively, outperforming translations generated individually for each source sentence by using M2M100, a general translation model. Furthermore, we also confirmed that the hub text found with our method generalizes across multiple language pairs such as Ja–En and De–En.

First place in the English-Japanese task
The NTTSU team′s submission leverages several large language models developed through a training procedure that includes continual pre-training and supervised fine-tuning. For paragraph-level translation, we generated synthetic paragraph-aligned data and utilized this data for training.In the task of translating Japanese to Chinese, we particularly focused on the speech domain translation. Specifically, we built Whisper models for Japanese automatic speech recognition (ASR). We used YODAS dataset for Whisper training. Since this data contained many noisy data pairs, we combined the Whisper outputs using ROVER for polishing the transcriptions. Furthermore, to enhance the robustness of the translation model against errors in the transcriptions, we performed data augmentation by forward translation from audio, using both ASR and base translation models.To select the best translation from multiple hypotheses of the models, we applied Minimum Bayes Risk decoding + reranking, incorporating scores such as COMET-QE, COMET, and cosine similarity by LaBSE.