DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answering

We prepare multiple dataset variants to isolate the contributions of each component in our pipeline.

Captioning:

• DEJIMA-Cap-Simple: filtered raw image–alt-text pairs (simple captions).
• DEJIMA-Cap-Refined: only LLM-refined alt-texts.
• DEJIMA-Cap-Detection: captions generated by the LLM from detection tags only.
• DEJIMA-Cap-All: integrates both alt-texts and detection tags as inputs to the LLM.

VQA:

• DEJIMA-VQA-Refined, DEJIMA-VQA-Detection, and DEJIMA-VQA-All.

Table: Statistical comparison of Japanese V&L datasets (counts, averages, and vocabulary sizes).

To examine DEJIMA’s representational coverage relative to existing datasets, we analyzed 2D feature distributions by applying PCA to CLIP image embeddings. All datasets were jointly projected into a shared 2D space and discretized on a common 60×60 grid (with 2% padding) to obtain probability maps pd(i,j).

For each dataset, we computed two measures: (1) the asymmetric coverage rate Coverage(P|Q)=∑_b∈occQ p_P(b), quantifying how much probability mass of P lies in bins occupied by Q; and (2) the bidirectional KL divergences KL(P||Q) and KL(Q||P) (with ε=1e−12) to capture overlap and distributional divergence in the shared space.

Using the domestic dataset recruit-jp as the reference target, DEJIMA achieved the highest coverage Coverage(target|DEJIMA)=0.785, exceeding Japanese Visual Genome (0.435), STAIR Captions (0.430), MS COCO (0.406), and GQA (0.342). This indicates DEJIMA spans about 79% of the visual domain occupied by real Japanese imagery. Conversely, for Coverage(dataset|target), Japanese Visual Genome was highest (0.534), followed by MS COCO (0.502) and STAIR (0.492), while DEJIMA was lower (0.192), suggesting broader support beyond the domestic domain.

KL divergences showed a consistent pattern: KL(recruit-jp||DEJIMA)=6.03 (lowest), Japanese Visual Genome (≈12.2), STAIR (≈12.3), MS COCO (≈12.8), GQA (≈14.2). In reverse, KL(DEJIMA||recruit-jp)=16.4, indicating DEJIMA includes regions not present in recruit-jp.

PCA projection of CLIP image embeddings: DEJIMA covers the domestic region (recruit-jp) and extends smoothly toward broader global contexts.

We evaluated caption and VQA quality via pairwise human comparisons (150 samples per dataset, randomized order, 80 crowd workers). We inserted control items for quality and removed inconsistent annotations; final sample sizes (n) are shown in each table.

Caption metrics: Japaneseness of image, Japaneseness of text, Naturalness of text, Image–text consistency, Coverage, Expressiveness. VQA metrics: Japaneseness (image/text), Naturalness of text, Q–A relevance, Q–image consistency, Answer correctness.

Significance was tested with two-sided binomial tests against 50%, using the 5% level with Holm–Bonferroni correction. A ^* indicates significance at 5%.

Table: Evaluation on the learned model using Japanese V&L benchmarks.

Figure: Sample output from the trained VLM (LLaVA).

Human Evaluation of VLM Outputs (Heron-bench)

In addition to automatic evaluations, we conducted human evaluations of VLM outputs on Heron-bench. All outputs for 103 questions were assessed following the same protocol as the dataset-level evaluation. Each question’s pair of model outputs (randomized order) was compared by one Japanese-speaking worker. In total, 412 samples (103 × 4 comparisons) were evaluated by 28 workers. One control item per worker was used; inconsistent workers (≈21.4%) were excluded. The effective sample size (n) for each comparison is shown below.

Metrics: Japaneseness of text, Naturalness of text, Q–A relevance, Answer correctness. Significance was tested with two-sided binomial tests against 50%, at the 5% level with Holm–Bonferroni correction. ^* indicates significance at 5%.