Downey

Jonathan Downey

Measurement Scientist • Psychometrician • AI Evaluation

jdowneyphd@gmail.com

www.github.com/JonathanDowney

www.officehours.courses


Skills

Measurement & Validation: argument-based validity frameworks; psychometric modeling (IRT/Rasch; multilevel); measurement invariance & construct stability; mixed-method validation design.

Model Evaluation: end-to-end evaluation pipeline design; cross-LLM benchmarking/model comparison; experiment tracking; reliability, transfer, and uncertainty estimation; Monte Carlo, bootstrap, and power analysis workflows.

Statistical Computing: R, Python, STAN; reproducible and scripted data processing; large-scale batch workflows.

Software & Infrastructure: PostgreSQL schema design; REST APIs and microservices; Docker; AWS; CI/CD (GitHub Actions); analytics dashboards; unit testing and static type checking.

Online Learning: Moodle, Canvas, Gradescope.


Languages

English: Native speaker (C2+)

Spanish: Fluent (C2)

French: Advanced (B2)

German: Intermediate (B1)

Basque, Arabic (Moroccan): Advanced beginner (A2)


Downey
PSY31, Fall 2023, Tufts University

Downey
Teaching technical writing at King Faisal University, Saudi Arabia

Downey
Teaching academic English in Al-ahsa, Saudi Arabia

Downey
Solving a tricky problem...

Downey
Planting trees at UCSB with grad residence assistants

Downey
Public speaking workshopping session... a great group!

Downey
David Bowie from Labyrinth visiting UCSB on Halloween :)

Downey
Painting in the style of the Old Masters...

Downey
Hanging out with Greta, a very good dog

Downey
Mendocino County, California

Downey
Sunset at UCSB

About Jonathan

Jonathan Downey is a postdoctoral measurement scientist and educational research scholar with New Meridian Corporation through Carnegie Mellon University’s Enhance Fellowship. He holds a PhD in Education (Quantitative Methods) from the Gevirtz Graduate School of Education at the University of California, Santa Barbara. His research focuses on argument-based validity, psychometrics , and the evaluation of large-scale assessments. He currently leads and a statewide validation study of Montana's MAST through-year assessment system, integrating statistical, survey, and qualitative evidence to offer actionable public policy recommendations.

In addition to large-scale assessment research, his work examines how AI-assisted tools can support measurement tasks such as item classification and reporting, and what evidence is required to justify their use across contexts. He previously studied Applied Linguistics at the University of the Basque Country (M.A.). He has taught statistics, ESL, and high school Spanish in both domestically and internationally.

Education

University of California Santa Barbara
May 2023

PhD in Education: Quantitative Methods in the Social Sciences


University of the Basque Country / Euskal Herriko Unibersitatea
March 2013

MA in Language Acquisition in Multilingual Settings


Tufts University
May 2009

BA in Economics, BA in Spanish Literature

Work Experience

Research Scientist and Enhance Program Fellow
New Meridian Corporation / Carnegie Mellon University
Fall 2024-current
  • Lead statewide validation research for Montana’s through-year assessment (grades 3–8), integrating psychometric, survey, and qualitative evidence into structured validity arguments.
  • Analyze large-scale student and educator data (35,000+ students) and communicate findings to the Montana Office of Public Instruction and vendor teams.
  • Co-design and prototype software systems integrating LLMs for automated item classification, construct-aligned item generation, and reporting features.
  • Build evaluation pipelines and cross-model benchmarking workflows; calibrate automated outputs against expert-generated gold-standard labels.
  • Develop internal dashboards and reproducible workflows to support iterative evaluation, reporting, and stakeholder decision-making.

Visiting Scholar
Tufts University: Education Department
Spring 2024
  • Engaged in research and instructional collaborations with department faculty.

Statistics Instructor
Tufts University: Psychology Department
Fall 2023
  • Held interactive lecture sessions, lab sections, and office hours consultation for 150 students.
  • Designed lesson materials and assessments based on real-world statistics applications.
  • Coordinated an instructional team of six teaching assistants.

Instructional Design Consultant
UC Santa Barbara, Gevirtz Graduate School of Education
Sept 2017 - June 2023
  • Conducted qualitative research targeting improvements to GauchoSpace, the campus online learning platform.
  • Designed and presented professional training workshops.
  • Provided in-person and online instructional technology support for university faculty.
  • Drafted and edited technical and training documentation.

Information Technology Liaison
UC Santa Barbara, Gevirtz Graduate School of Education
Jan 2017 - June 2023
  • Consulted with faculty and students regarding GGSE educational technology policy and budget.
  • Coordinated graduate student communication with the GGSE IT Services Office.

Project Manager / Researcher: Visual Feedback Lab
UC Santa Barbara: Computer Science and Electrical Engineering Depts.
Sept 2017 - June 2018
  • Created and managed a collaboration between undergraduate Computer Science students, the UCSB Linguistics department, and the Gevirtz Graduate School of Education to design language education software.

Researcher: Arbis Pictus Lab
UC Santa Barbara: Computer Science Department
Dec 2016 - Oct 2018
  • Co-organized an experiment to investigate the potential pedagogical benefits of augmented reality user interfaces and conducted quantitative data analysis of participant data.

Researcher/Course Designer: Basque 154W (Online Course)
UC Santa Barbara: Spanish and Portuguese Department
March 2017 - June 2018
  • Conducted 20 hours of audio and video interviews for language learning resources.
  • Conducted qualitative thematic organization of interview resources.

Spanish Teacher
Arrupe Jesuit High School
Sept 2015 - June 2016
  • Taught elementary Spanish language and Spanish literature to high school juniors.
  • Created effective task-based syllabi and lesson plans.

Academic Writing Instructor
King Faisal University, Engineering Department
Sept 2015 - June 2016
  • Taught university students literary organization skills and writing strategies.
  • Created syllabi and lesson plans.

Director of Programs: Spain and Morocco
Walking Tree Travel
July 2012 - July 2014
  • Designed and coordinated educational travel programs in Spain and Morocco.
  • Coordinated local teachers, tour guides, and vendors.
  • Managed program budget and finances.

Instructor: English for the Metal-working Sector, English for the Wine-tourism Sector
LANBIDE (Basque Department of Employment)
Jan 2013 - Dec 2013
  • Taught English conversation tailored to specialists in the metal-working and wine-tourism sectors.

Elementary School Teacher
Denver Montclair International School
Sept 2011 - August 2012
  • Taught language and athletics classes for children ages 3-4 and 5-10.
  • Developed lesson plans and coordinated work with teaching assistants.

Freelance Interpreter/Translator
Adams 12 and Adams 50 school districts, Thornton/Westminster, CO
Sept 2009 - Aug 2011
  • Spanish <> English Interpreter
  • English > Spanish Translator

Awards

  • 2022 Morrison Fellow (Gevirtz Graduate School of Education, UC Santa Barbara)

  • 2020 Data Scholarship (University of Pennsylvania Linguistic Data Consortium)

Publications

  • Inferential Validity of PISA’s measure of SES (Manuscript in preparation)

  • ARbis Pictus: A Study of Vocabulary Learning with Augmented Reality - in iEEE Transactions on Visualization and Computer Graphics (November 2018)

  • Supporting content for Language Learning and Technology Journal (Winter 2018)

Conference Presentations

AERA Research In-Progress Gala
March 2021
"A Rasch-based Approach to Validating Essay Ratings"
International Objective Measurement Workshop (IOMW) at UC Berkeley
Feb 2021
"Measuring Productive Vocabulary Using Item Response Theory"
CATESOL Los Padres conference at UCSB
Feb 2020

"Hidden Dangers in L2 Assessment: Issues of Reliability and Validity"


Foreign Language Education Symposium (FLEDS) conference at Middlebury Institute of International Studies at Monterey (MIIS)
Nov 2018

"Encouraging Learner Engagement with Peer Feedback"


UCSB Verbal Kaleidoscope Conference
April 2018

"Sociocultural Strategies and Digital Tools for Minority Language Instruction"


UCSB and GGSE “GradSlam” events
April 2017

"Visual Feedback: An Easier Way to Improve Pronunciation"


Memberships / Organizations

National Council for Measurement in Education (NCME): Member
Sept 2019 - Current

GGSE IT Committee: Board member
Dec 2016 - May 2023