Model Card

Yotsuba is an improved re-implementation of the original GPT-4chan's concept, built on the modern Qwen 2.5 architecture. It's continually pretrained on over 2 million threads from 4chan's /pol/ with support for structured output.

The dataset used is a cleaned version of our 4chan-pol dataset, which itself is a reformatted version of the Raiders of the Lost Kek dataset, containing 3.5 years worth of data from 4chan's /pol/ board.

Yotsuba can output in a structured pseudo-XML style format intended to make it easy for applications or frontends to parse.

Weights are available in the Hugging Face repository under strict usage restrictions.

Disclaimer & Usage Restrictions

This model is trained on data that is known to contain offensive, disturbing, explicit, hateful or otherwise objectionable content. The outputs generated do not reflect the views or opinions of the developers and are intended solely for research and experimentation.

We explicitly prohibit usage, deployment, or distribution of the model or its outputs in any manner that exposes its generated outputs to individuals who have not consented to receiving them. This includes, but is not limited to:

• Posting generated outputs in a public space, be it manually or with automation.
• Sending generated outputs to anyone without their prior consent.
• Deploying the model to communicate with individuals without their prior consent.
• Sending the model weights or its associated quants to anyone without their prior consent.

Obtaining consent requires you to ensure all parties are aware of and acknowledge the above disclaimer before the action.

These restrictions are described in more detail in the ATTACHMENT A: USE-BASED RESTRICTIONS section of our OpenRAIL license.

Model Performance

/pol/'s distinct tone, slang, topics, and formatting idiosyncrasies like greentexting are consistently replicated.

Benchmark tasks were chosen based on Qwen 2.5's technical paper. Coding tasks were omitted as it was deemed out of scope.

Task	n-shot	Metric	Qwen 2.5 7B	Yotsuba 7B	Delta (Δ)
General Tasks
mmlu	5	acc	74.25	72.44	-1.81
mmlu_pro	5	exact_match	48.71	44.37	-4.35
mmlu_redux_generative	5	exact_match	49.87	49.32	-0.54
bbh	3	exact_match	69.10	65.14	-3.96
arc_challenge	25	acc_norm	63.99	61.18	-2.82
truthfulqa_mc2	0	mc2	56.33	48.54	-7.79
winogrande	5	acc	75.93	74.03	-1.89
hellaswag	10	acc_norm	80.28	78.12	-2.16
Math & Science Tasks
gpqa_main_n_shot	5	acc	36.28	37.50	+1.22
gsm8k	4	exact_match	79.83	75.89	-3.94
minerva_math	4	exact_match	36.30	27.78	-8.52
Safety & Bias Tasks
bbq	0	acc	49.32	47.49	-1.83
crows_pairs	0	pct_stereotype	66.49	68.87	+2.39
ethics_cm	0	acc	63.45	64.68	+1.24

A notable result is TruthfulQA's large drop (-7.79 points). It implies that exposure to unverified internet conspiracy and satire degrades fact-selection. This is contrasting to GPT-4chan's benchmark data, which showed an increased TruthfulQA score compared to its base GPT-J 6B model.

Benchmarks were run using lm-eval version 0.4.12 with vLLM as the inference backend. Results are not comparable to Qwen's official numbers in their report as they may use a different scoring/evaluation method.

The Image Problem

A large number of threads from /pol/ have images in context, be it the main post or following replies. Since Yotsuba was trained purely on text, these threads don't make sense without the images in context.

We didn't remove these threads in order to keep the corpus huge, but this results in the model learning a lot of discussions that it can't make sense of, and is probably a strong source of coherence drops in its generations.

Structured Output

Yotsuba can output in a structured pseudo-XML style format that allows easy parsing of individual posts and reply references within a thread:

<thread>

<post=1>
...

<post=2>
...

<post=3>
>>2
...a reply to post 2...

</thread>

No special tokens are required.

A full thread can be generated with the prompt:

<thread>

<post=1>

or with a starting post:

<thread>

<post=1>
...

<post=2>

Posts do not have closing </post> tags like </thread>.

Replies follow 4chan's >>[id] syntax, linking to the <post=x> post ID.

The model cannot be forced to reply to a specific post in the middle of a thread. If a reply is required for a specific piece of text, it has to be the opening <post=1> post.

Acknowledgements & Citations

The dataset used to train this model is derived from the work of Papasavva et al. and the original Raiders of the Lost Kek dataset. Please refer to the original publication for information about the source dataset, collection methodology, and research context.

https://arxiv.org/abs/2001.07487

The base model used is Qwen 2.5 7B. Please refer to their technical report for more information about the Qwen 2.5 family of models.

https://arxiv.org/abs/2412.15115

@misc{papasavva2020arxiv,
  author        = {Papasavva, Antonis and Zannettou, Savvas and De Cristofaro, Emiliano and Stringhini, Gianluca and Blackburn, Jeremy},
  title         = {Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board},
  year          = {2020},
  eprint        = {2001.07487},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CY}
}

@misc{qwen2.5,
    title = {Qwen2.5: A Party of Foundation Models},
    url = {https://qwenlm.github.io/blog/qwen2.5/},
    author = {Qwen Team},
    month = {September},
    year = {2024}
}