Inside Amazon Connect Talent: The science of structured AI interviews 

For organizations hiring hundreds or thousands of people, every interview presents the same challenge: How do you give candidates a consistent opportunity to demonstrate their skills while gathering the evidence needed to make informed hiring decisions?  Structured behavioral interviews offer a research-backed approach to this challenge. They use consistent, job-related questions and clear scoring criteria […]

Nov 19, 2026 - 22:00
 2
Inside Amazon Connect Talent: The science of structured AI interviews 

For organizations hiring hundreds or thousands of people, every interview presents the same challenge: How do you give candidates a consistent opportunity to demonstrate their skills while gathering the evidence needed to make informed hiring decisions? 

Structured behavioral interviews offer a research-backed approach to this challenge. They use consistent, job-related questions and clear scoring criteria to give candidates an opportunity to demonstrate their skills. Decades of research in industrial-organizational (I-O) psychology have shown that structured interviews help organizations evaluate candidates and predict job performance (Schmidt & Hunter, 1998; Sackett et al., 2022). Yet delivering structured interviews at scale requires significant time and expertise to develop competencies, create questions, train interviewers, and evaluate responses consistently. 

Amazon Connect Talent brings established hiring science to AI-led interviews at scale. Connect Talent uses structured questions, adaptive follow-ups, and evidence-based scoring to help organizations gather consistent, job-related information while giving more candidates the opportunity to demonstrate their strengths.

We built Connect Talent around three core design principles and evaluated how well the service applies them through a combination of internal research and independent expert review. The findings offer a closer look at the science behind Connect Talent’s AI-led, structured interviewing. 

Three core design principles

1. Build interviews around job-related evidence

Connect Talent builds each interview around the competencies required for the role. The service uses job requirements and established occupational frameworks to create behaviorally anchored scorecards. Each scorecard defines specific strength and concern signals, and every interview question targets the behaviors it aims to uncover. 

The AI interviewer also adapts to each response. After every answer, it checks whether the candidate has provided enough evidence for the competency being assessed. When details are missing, it asks targeted follow-up questions. This approach preserves the structure of the interview while giving candidates more opportunities to explain their experience. 

2. Separate interviewing from scoring

Connect Talent separates the interview conversation from the evaluation. The AI interviewer gathers evidence but does not assign scores. Afterward, a separate evaluation process identifies which behaviors the candidate demonstrated and cites the supporting transcript passages. 

The service calculates each competency score from 1 to 5 using a fixed formula. When the transcript lacks sufficient evidence, Connect Talent flags the competency instead of producing a score. Hiring teams can review the evidence behind each rating, and missing evidence does not automatically count against the candidate. 

3. Build fairness protections into the design

Connect Talent evaluates transcribed candidate responses without using video, voice tone, or physical appearance to calculate competency scores. The service also excludes candidate identity information from scoring and evaluates each competency separately. These safeguards help focus assessments on job-related behaviors and limit the influence of unrelated information. 

What the research shows

To understand how well these design principles work in practice, we combined independent I-O psychology evaluations with internal research. We examined whether the service generates job-relevant interview content, identifies behavioral evidence in line with expert assessments, gathers more complete responses through follow-up questions, and distinguishes between levels of demonstrated competency.

Connect Talent keeps interview questions focused on job-relevant skills 

The quality of a structured interview starts with the skills it measures. Candidates need questions that reflect the requirements of the role, and hiring teams need confidence that the interview assesses those skills. 

Amazon engaged an independent I-O psychology firm to evaluate Connect Talent’s AI-generated interview content. The firm reviewed the service’s baseline output before human editing and examined how well its competencies, questions, and rubrics aligned with job requirements. The evaluation found that Connect Talent retained critical job information in 100% of the job descriptions reviewed. Of the competencies generated, 91% were clearly traceable to the job description. The firm’s psychologists also found that 98% of interview questions targeted the intended competencies, and 94% of rubric signals clearly measured the intended competency. 

These findings show that Connect Talent generated job-relevant interview content that met independent expert criteria in a vast majority of cases. Hiring teams can then review, select, and edit the competencies, scorecards, and questions before using them. This human oversight gives organizations control over the interview process while helping them apply structured hiring practices at scale. 

Connect Talent’s candidate scoring matched independent expert decisions more than 89% of the time 

Relevant questions provide the foundation for a structured interview. The next challenge is evaluating the evidence candidates provide with consistency.  

To examine this, an independent I-O psychology firm reviewed 36 interviews assessing whether candidates demonstrated specific strengths and concerns, creating 252 rubric signals.  

Connect Talent evaluated the same transcripts, reviewed hundreds of rubric signals, and aligned with expert scoring decisions in 89.3% of cases. Scoring alignment reached 97.6% when candidates provided clear evidence and 92.9% when only considering candidate concern signals, which identify behaviors that may indicate a poor fit for the intended role.  

The evaluation also produced a Cohen’s kappa of 0.79, a statistical measure that eliminates random chance coincidence. A Cohen’s kappa of 0.79 is just under the standard of 0.81 for almost perfect agreement. This outcome indicates substantial agreement between Connect Talent and the independent expert candidate scoring.  

Follow-up questions give candidates more opportunities to demonstrate their strengths 

Candidates may have relevant experience but leave important details out of their first response. In a traditional one-way video interview, a candidate may receive a fixed question and have only one opportunity to answer it. Hiring teams must then evaluate the information available, even when the response leaves many important questions unanswered. 

Connect Talent uses adaptive follow-ups to gather more complete evidence. For example, a candidate might describe improving a slow system. The AI interviewer can ask what changed, which actions the candidate took, and how they measured the improvement. These questions help candidates explain their contributions and results in greater detail. 

We studied the impact of follow-up questions by comparing assessments based on a candidate’s first response with assessments based on the full interview conversation. In a preliminary evaluation, scores based only on the first response were up to 1.14 points lower on a five-point scale than scores based on the full conversation. Assessments using the full conversation also came closer to expert evaluations. 

Without follow-up questions, the scoring process identified fewer strength signals and more concern signals. In some cases, the candidate’s initial response simply lacked the information needed to demonstrate a relevant behavior. The full conversation captured more of what candidates demonstrated and gave hiring teams a more complete record of their experience. 

Structured scoring helps hiring teams recognize meaningful differences 

Hiring teams need scores that reflect meaningful differences in demonstrated skills. Yet research has found that some AI scoring approaches concentrate ratings within a narrow range, making those differences harder to recognize (Zhu et al., 2026). 

Connect Talent uses behaviorally anchored scorecards to evaluate specific strength and concern signals. Rather than asking an AI model to assign an overall rating, the service identifies evidence for each behavior and calculates the competency score using a predefined formula. 

Consider the competency “collaboration”. A general AI scoring approach might evaluate a candidate’s overall collaboration skills based on the interview transcript. Connect Talent examines specific behaviors, such as whether the candidate sought input from colleagues before making a decision. Each finding contributes to the final score. 

We evaluated this approach across hundreds of expert-rated interview assessments. Connect Talent’s structured scoring used all five points of the rating scale, while holistic LLM-as-judge scoring concentrated its ratings within the middle three points. 

This wider distribution allows the scoring system to represent more levels of demonstrated competency. The behaviorally anchored scorecards also help hiring teams understand the evidence behind each rating. 

Bringing hiring science to AI-led interviews 

Connect Talent brings the rigor of structured interviewing to more people while giving each candidate a meaningful chance to demonstrate their skills. The result is a new opportunity for organizations hiring at scale: more candidates can show what they can do, and more hiring teams can make decisions grounded in the evidence that matters. 

Learn More at AWS re:Invent 2026 

Join us at AWS re:Invent 2026 for BIZ332: Every Candidate, Every Time: AI-Led Structured Interviews with Amazon Connect Talent, an interactive Chalk Talk led by Sunitha Srivatsa and Gerald Friedland. 

The session will explore the science behind Connect Talent, including structured interviewing, adaptive follow-up questions, and evidence-based scoring. 

Session 1: Monday, November 30, 2026, 10:00–11:00 AM, Caesars Forum – Level 1, Academy 409. 

Repeat session: Thursday, December 3, 2026, 3:00–4:00 PM, MGM Grand, Premier 309. 

Register for AWS re:Invent 2026 

References 

Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. https://doi.org/10.1037/apl0000994 

Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research. Psychological Bulletin, 124(2), 262–274. 

Booth, B. M., Hickman, L., Subburaj, S. K., Tay, L., Woo, S. E., & D’Mello, S. K. (2021). Bias and Fairness in Multimodal Machine Learning: A Case Study of Automated Video Interviews. Proceedings of the 2021 International Conference on Multimodal Interaction, 268–277. https://doi.org/10.1145/3462244.3479897 

Zhu, K., Sheng, Z., Liu, Z., & Tan, J. (2026). Scale Collapse, Model Disagreement, and False Precision: A Multi-Model Audit of LLM Behavioral Interview Scoring.  


About the authors

Sunitha Srivatsa

​​Sunitha Srivatsa is a Principal Software Development Engineer at AWS and the technical lead for Amazon Connect Talent and Amazon Connect Customer Global Resiliency. Amazon Connect Talent is an agentic AI hiring solution that uses AI-led structured interviews and behavioral evidence to help businesses hire faster and more fairly at scale. Sunitha owns the end-to-end architecture for Connect Talent, with a focus on the AI interviewer. On Global Resiliency, she leads the design of multi-region capabilities that let customers run their contact centers across multiple AWS Regions in an active-active setup. Sunitha joined Amazon in 2013 and has worked on Amazon Connect since 2020. Before that, she built products across AWS, including the Amazon Silk browser and Alexa for Business. Her specialties include system architecture, multi-region resiliency, and building services that businesses can rely on at scale. Outside work, Sunitha enjoys traveling the world with her two young kids and dabbles in yoga and dance.​ 

Gerald Friedland

Gerald Friedland is a Principal Applied Scientist at AWS and the science head of Amazon Connect Talent. Amazon Connect Talent is an agentic AI hiring solution that uses AI-led structured interviews and behavioral evidence to help businesses hire faster and more fairly at scale. Gerald leads the science behind the service’s high-volume AI interview systems. He also created privacy-enhanced synthetic data for AWS Clean Rooms and co-authored MITRA, Amazon’s tabular foundation model. He has more than 25 years of experience in research, engineering, and teaching. Before Connect Talent, he led science work on Amazon SageMaker. He also founded Brainome, an AutoML company, and Audeme, which built a speech recognition board for Arduino. Earlier, he was adjunct faculty at UC Berkeley, Principal Data Scientist at Lawrence Livermore National Laboratory, and a research director at the International Computer Science Institute. His specialties include information theory, machine learning, and explaining AI through first principles. He has written four books, including Information-Driven Machine Learning, and more than 200 publications, and holds a PhD in computer science from Freie Universität Berlin.​ 

Jad Mokdad 

​​Jad Mokdad is a Senior Software Development Engineer on the Applied AI Solutions team at AWS, where he leads engineering work on Amazon Connect Talent. Amazon Connect Talent is an agentic AI hiring solution that uses AI-led structured interviews and behavioral evidence to help businesses hire faster and more fairly at scale. Jad helps build the systems that power the service end to end. He has more than 12 years of industry experience, including eight years at Amazon. Before Connect Talent, he helped build systems that support Amazon’s high-volume hiring operations worldwide, including hiring workflow platforms, virtual hiring experiences, and generative AI assistants for candidates. His specialties include hiring technology and reducing repetitive recruiting tasks so hiring teams can spend more time with people. Outside work, Jad enjoys boxing, weightlifting, cooking, puzzles, and spending time with his two French bulldogs.​ 

Guru Nayak 

​​Guru Nayak is a Senior Applied Scientist at AWS and the technical lead for the science behind Amazon Connect Talent. Amazon Connect Talent is an agentic AI hiring solution that uses AI-led structured interviews and behavioral evidence to help businesses hire faster and more fairly at scale. Guru is responsible for the service’s scientific approach: how the AI gathers behavioral evidence during an interview, how that evidence becomes an evaluation, and what must be proven before a method ships. He has nearly seven years of experience as an applied scientist at Amazon. Before Connect Talent, he worked on Amazon SageMaker and on generative AI conversational ad experiences on Amazon. He holds a PhD in computer science from the University of Minnesota, where he worked with NASA scientists to apply machine learning to environmental problems, such as mapping forest fires from satellite imagery. His specialties include machine learning, AI evaluation, and applied research. Guru lives in Seattle and volunteers with a local animal rescue.​ 

Martin Yu

​​Martin Yu is a Senior Research Scientist on the Global Hiring Science team in Amazon’s PXT organization, where he leads the science behind Amazon’s hiring assessments. Amazon Connect Talent is an agentic AI hiring solution that uses AI-led structured interviews and behavioral evidence to help businesses hire faster and more fairly at scale. For Connect Talent, Martin advises on making its assessments and AI-led interviews meet established measurement standards and follow scientific best practices. Before joining Amazon, he spent seven years building assessments and conducting hiring research for private-sector and government organizations. His specialties include assessment design, hiring research, and measurement science. He holds a PhD in industrial-organizational psychology from the University of Minnesota.​ 

Ikkei Itoku 

​​Ikkei Itoku is an Applied Scientist on the Agentic AI team at AWS and a founding scientist for Amazon Connect Talent and Amazon Quick. Amazon Connect Talent is an agentic AI hiring solution that uses AI-led structured interviews and behavioral evidence to help businesses hire faster and more fairly at scale. Ikkei helps the service shorten enterprise hiring cycles from weeks to days. He has more than 15 years of experience building AI and data systems. Before his current role, he worked on talent analytics science at Amazon, where he was lead author of a KDD 2024 paper. Earlier, he led a workforce analytics data science team at JPMorgan Chase and was the first data scientist on the People Analytics team at Bloomberg. His specialties include multi-agent systems, large language model evaluation, voice-based AI agents, and fairness in hiring models. He holds a Master of Engineering in data science from Columbia University.​ 

Minje Choi 

​​Minje Choi is an Applied Scientist on the Applied AI Solutions team at AWS, where he helps develop the science behind Amazon Connect Talent. Amazon Connect Talent is an agentic AI hiring solution that uses AI-led structured interviews and behavioral evidence to help businesses hire faster and more fairly at scale. Minje leads work on how the service generates job-related competencies, scorecards, and interview questions, including designing and testing the algorithms behind each step. He also contributes to research on how the AI interviewer conducts structured interviews. Before Connect Talent, he helped design personalized shopping recommendations at Amazon. Earlier, he completed a PhD in information at the University of Michigan and a postdoc at Georgia Tech, where he researched how to evaluate large language models on social factors. His specialties include large language model evaluation, natural language processing, and structured interview design.​

Jat AI Stay informed with the latest in artificial intelligence. Jat AI News Portal is your go-to source for AI trends, breakthroughs, and industry analysis. Connect with the community of technologists and business professionals shaping the future.