LockedIn AI is the #1 real-time AI interview and meeting copilot, trusted by over one million users worldwide. We are building a next-generation career preparation platform powered by real-time AI assistance.
Our product helps users perform better in live interviews, coding assessments, and professional meetings by delivering intelligent, context-aware AI support at scale.
Role Overview
We are looking for a detail-driven AI Analysis Specialist to measure, evaluate, and improve the performance of LockedIn AI’s AI systems across all key dimensions. This role sits at the intersection of data, AI evaluation, and product intelligence. You will analyze how our models perform in real-world usage, identify failure patterns, and turn insights into actionable improvements that enhance product quality for over 1 million users. You will be responsible for building evaluation systems, monitoring AI behavior in production, and connecting model performance to user outcomes and business impact. The ideal candidate combines strong analytical skills with a deep understanding of how AI systems behave in production environments.
Key Responsibilities
• AI Model Evaluation & Quality Analysis
• Design frameworks to evaluate AI accuracy, relevance, latency, and hallucination rates
• Build benchmark datasets, scoring rubrics, and golden test sets
• Identify model failure modes and performance regressions
• Evaluate RAG pipeline quality and grounding accuracy
• Data Analysis & Insights
• Analyze large-scale datasets from user interactions and AI outputs
• Build automated workflows for ongoing performance monitoring
• Perform exploratory analysis to uncover trends and insights
• Run A/B tests and statistical evaluations of AI improvements
AI Monitoring & Reporting
• Build dashboards for real-time AI system health tracking
• Monitor drift, latency issues, and quality degradation
• Track cost metrics including token usage and model routing efficiency
• Create executive reports with actionable insights
Product & User Impact Analysis
• Measure impact of AI features on engagement, retention, and satisfaction
• Define and track KPIs for new AI features
• Analyze user behavior patterns and AI performance by segment
• Support experiment design and evaluation strategies
Data Quality & Pipeline Support
• Improve dataset quality for evaluation and training
• Define annotation guidelines and validation rules
• Collaborate on data pipelines and analytical infrastructure
• Ensure data accuracy and consistency across systems
Required Qualifications
• 3+ years in data analysis, AI evaluation, or product analytics
• Experience evaluating LLM or ML systems in production
• Strong background in A/B testing and statistical analysis
• Experience with AI monitoring or observability systems
• Startup or fast-paced environment experience preferred
Education
Bachelor’s degree in Data Science, Statistics, Computer Science, Mathematics, or related field (or equivalent practical experience).
Technical Skills
• Strong Python and SQL proficiency
• Experience with pandas, NumPy, and visualization tools
• Familiarity with LLM evaluation metrics and frameworks
• Knowledge of A/B testing and statistical methods
• Experience with BI tools (Looker, Tableau, Metabase, etc.)
• Familiarity with cloud data warehouses (BigQuery, Snowflake, etc.)
• Exposure to AI observability tools (Langfuse, W&B, Arize, etc.)
Soft Skills
• Strong analytical thinking and structured problem-solving
• Ability to translate data into clear business insights
• Strong communication skills for technical and non-technical audiences
• Ownership mindset with end-to-end responsibility
• Collaborative approach across engineering and product teams
Preferred Qualifications
• Experience with real-time AI systems or conversational AI
• Background in NLP evaluation or dialogue systems
• Familiarity with LLM-as-a-judge evaluation methods
• Experience optimizing token usage and model costs
• Experience in edtech, career tech, or SaaS B2C products
• Contributions to analytics or evaluation frameworks
Pay/Wages | $110,000 – $165,000 USD/year
How to Apply? | https://www.lockedinai.com/careers/ai-analysis-specialist