I Tested 71 AI Model Configurations On My Own News Pipeline. The Best One Isn’t The One I Run.

🔍 Read the full analysis: I Tested 71 AI Model Configurations On My Own News Pipeline. The Best One Isn’t The One I Run. on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says testing 71 AI model configurations on his news-writing pipeline found GPT-6 Astra produced the highest-rated articles, but he selected the much cheaper GPT-6 Luna for production. In results published October 13, Meyer reported Astra scored 4.00 out of 5 in the initial English and German tests, while Luna scored about 3.67 and cost roughly one-hundredth as much per article.

Meyer, writing on ThorstenMeyerAI.com, tested configurations between September 8 and October 13, 2026, using his production prompt and real news sources. The test included API, subscription, free-tier and locally run open-weight models. Each source was turned into an English and a German article, then evaluated by two AI judges, GPT-6.1 Sol and Claude Opus 5.5. The judges assessed source fidelity, fabrication, clarity about evidence, news value, structure and polish, with German language quality included for that edition.

The initial test used two sources: a 6,340-character product filing and a 619-character political news item. Meyer said short sources make up much of his work, so the shorter item was particularly relevant. Stronger configurations also wrote 10 additional articles, mostly from short sources. Meyer treated this robustness run as a more reliable guide than small differences in the two-article comparison.

In that follow-up, Meyer reports Astra averaged 3.93 across 12 articles, with a 95% bootstrap interval of 3.83–4.02. Luna averaged 3.67, with an interval of 3.59–3.76. Astra cost about $0.12 per article and took around 50 seconds; Luna cost about $0.0012 and took around 17 seconds. Meyer says Luna’s source-fidelity score in the robustness run was 4.71 out of 5 and that all its articles produced valid output for his pipeline.

Cost and Reliability Shape the Choice

The results illustrate why a general model leaderboard may not identify the best option for a particular publishing task. Meyer’s decision depended not just on which system received the highest writing score, but on whether the quality difference justified the cost and delay at his publication’s volume. He says Astra’s stronger score was real in his test, but its roughly 100-times higher per-article cost led him to reject it for routine news production.

The findings also put attention on failure handling. Meyer reports that two models in his former backup chain scored poorly or failed to produce usable articles, with judges flagging invented quotations. A pipeline’s fallback behavior can affect published output when its main model is unavailable, so testing only the preferred model would miss a material source of risk. These are results from one publisher’s task and setup, not a universal ranking of AI systems.

A Test Built Around Real Articles

Meyer framed the work as a practical comparison rather than a general intelligence contest: the question was which configuration could turn actual sources into articles under the prompt his sites use. He applied a production-style validity check, counting an article only if it yielded valid output after the same repair step used in his pipeline. The evaluation therefore combines editorial judgments with a basic operational requirement.

His report says he considers differences under about 0.3 points in the initial two-article test potentially attributable to noise. The follow-up articles and reported confidence intervals were used to compare leading configurations more cautiously. The test was conducted in both English and German, but its source sample was limited to two items in the initial round; the additional robustness set was described as 10 more articles, mostly based on short sources.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of the Model Comparison

The findings come from one publisher’s pipeline, prompt, language editions and set of sources. The report does not establish that Luna will perform similarly for other publishers, article genres, source quality or editorial standards. The initial comparison used only two sources, and the larger follow-up was still a relatively small sample; the reported intervals describe uncertainty within Meyer’s test, not performance across all possible news assignments.

The scores were assigned by AI judges rather than reported as assessments by human editors. Both judges saw the original source, but the report does not establish how their ratings compare with an independent human review. It also does not provide enough detail here to verify every configuration’s access conditions or reproduce all results. Meyer reports that GPT-4.1 nano generated invented quotes and local gpt-oss-120b produced no valid articles in four attempts; these are observations from his test, not claims that the models always behave that way.

Luna Remains in Production

Meyer says GPT-6 Luna remains the production choice, with reasoning effort set to none. His test found the highest setting took about 211 seconds per article and cost 1.43 cents, compared with 17 seconds and 0.12 cents at none, without a reliable quality gain. He also says he removed GPT-4.1 nano and gpt-oss-120b from the news route on October 7.

The report does not announce a new test date or a planned change to the route. The next useful check would be whether Luna continues to meet Meyer’s quality and output-validity standards on a larger, ongoing set of production sources, and whether the results hold as model versions or the pipeline change.

Key Questions

Which model scored highest in Meyer’s news-writing test?

GPT-6 Astra led the initial test, scoring 4.00 out of 5 in both English and German. It averaged 3.93 across the 12-article robustness run, according to Meyer.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna scored 3.67 in the robustness run and produced valid articles, while costing about $0.0012 per article. Astra scored higher but cost about $0.12 per article, roughly 100 times more.

Did higher reasoning effort improve the articles?

Meyer reports no reliable gain across six Luna reasoning settings. The highest setting cost about 12 times as much and took about 12 times as long as the no-effort production setting, while its scores remained within the test’s noise range.

What problems did the test find with backup models?

Meyer says GPT-4.1 nano received low source-fidelity ratings and judges flagged invented quotations. Locally run gpt-oss-120b produced zero valid articles in four attempts and also generated attributed quotations that were not in the source, according to the report.

Do these results establish the best model for all newsrooms?

No. They reflect one publisher’s prompts, sources, languages and evaluation method. Other newsrooms would need to test models on their own assignments and standards before drawing a similar conclusion.

Source: ThorstenMeyerAI.com