Subject-Matter Experts - AI Benchmark Question Writers & Reviewers

Full Job Description

1-StopAsia is building a native-language benchmark that tests how well today's leading AI models reason in Asian and European languages. We are looking for native-speaking academics and professionals to write expert-level exam questions that the best AI models cannot answer correctly, and to review questions written by others. Your work will show AI developers where their models still fall short in your language, culture and field.

Languages and domains we are hiring for

Language / Domains

• Thai - Law
• Filipino (Tagalog) - Literature (Poetry / Classics)
• Vietnamese - International Relations and Political Science; Philosophy (including non-Western); Psychology; Religion and Religious Studies
• Korean - Economics (Finance / Business / Trade); History; International Relations and Political Science; Law
• Standard Russian - Several humanities and social-science domains – tell us your field

You may apply for more than one domain if you have the expertise.

What you will do

As a Question Writer:

• Write original, exam-style questions in your native language in your domain.
• Write both short-form questions (multiple choice or one exact answer) and long-form questions (one connected reasoning task, graded against a rubric).
• Test each question on 5 AI models (GPT, Gemini, Claude Opus, Kimi, Muse Spark) using free accounts; we tell you which versions to use.
• A question is accepted when at least 3 of the 5 models answer it wrongly.
• Also write at least 5 "non-challenge" questions per domain that the models do answer correctly.
• For every question, provide the correct answer, the reasoning that leads to it, why the question is hard, and what the models got wrong, all translated into English.
• Record everything in our structured Google Sheets template.

As a Reviewer:

• Check each accepted question against five criteria: well formed, unambiguous, genuinely multi-step, correct answer, and an explanation that traces the reasoning.
• Comment on anything that fails and approve or return the question.

What makes a good question

• It needs several connected reasoning steps, not a single fact lookup or several unrelated questions bundled together.
• Every requirement (format, method, length) and all needed context are stated in the question itself; if it refers to a text, the text is included.
• It has one well-defined correct answer, which you have solved yourself exactly as written.
• It draws on knowledge rooted in sources, institutions and traditions of your language, not on translated textbook material.

Requirements

• Native speaker of Thai, Filipino, Vietnamese, Korean or Russian.
• University degree in one of the listed domains (Master's or PhD preferred), and/or professional, teaching or research experience in it.
• Strong written English, enough to translate your questions, answers and explanations accurately.
• Comfortable working with AI chat tools and Google Sheets.
• Precise, patient and able to work independently to a deadline.

Nice to have: exam or test-writing experience; university teaching; expertise in more than one domain; prior work in AI training data, evaluation or annotation.

Scope and timing

• Per domain: 20 accepted questions (10 short-form, 10 long-form), plus at least 5 non-challenge questions.
• One expert may cover one or several domains.
• Start as soon as possible after onboarding; all work completed by 2 November 2026.
• Fully remote, flexible hours.

Benefits

• Question Writers are paid in USD per accepted question, with a higher rate for long-form questions. Questions that do not meet the acceptance threshold are not paid; please take this into account.
• Reviewers are paid per reviewed question or hourly.
• Rates are agreed individually; tell us your expected rate in your application.

Application questions

• What is your native language?
• Which domain(s) can you write or review expert-level questions in?
• What is your highest degree, and in which field?
• How many years of professional, teaching or research experience do you have in that domain?
• Would you like to work as a Question Writer, a Reviewer, or both?
• Do you understand that writers are paid only for accepted questions (at least 3 of 5 AI models fail)? (Yes / No)
• Your expected rate in USD per accepted short-form and long-form question (reviewers: per question or per hour).
• How many accepted questions could you realistically deliver per week until 2 November 2026?
• Have you done AI training, evaluation or annotation work before? If yes, briefly describe it.
• Write one sample question in your domain (in your language, with an English translation and the correct answer) that you believe needs several reasoning steps.

To apply, submit your CV and answer the questions above instead of sending cover letter.

Qualifications And Requirements

Mid-Senior level

Mid-Senior level