Required skills & experience
We are looking for a strong technical engineer who can design evaluation strategies, implement validation frameworks, and turn regulatory requirements into automated, measurable quality gates.
- Strong technical background validating and testing complex systems, with hands-on experience building test harnesses.
- Experience building automated test frameworks, including creation of golden datasets, custom validation frameworks, and automated correctness/validation logic.
- Proven experience designing and implementing benchmarking strategies and quality metrics for complex outputs.
- Familiarity with LLM evaluation, validation techniques, and quality metrics (including how to interpret results and drive iteration).
- Proficiency with test automation tools, with Playwright preferred for web/HTML-level validation.
- Solid understanding of XBRL, XHTML, XML, or financial regulatory reporting (strongly preferred).
- Strong problem-solving skills and the ability to work with ambiguity and incomplete requirements while still converging on reliable quality criteria.
- Excellent communication and collaboration skills across R&D and production engineering stakeholders.
- Experience with financial or compliance testing and validation.
- Familiarity with SEC filing requirements and investment company regulations.
- Background in data quality validation and metrics design.
- Experience with CI/CD pipelines and DevOps practices to run benchmarks and validations consistently.
Soft skills we value: we expect clear written communication, structured thinking, ownership of quality outcomes, and an iterative mindset that balances compliance precision with practical engineering constraints.
Funciones
We will have you lead the design and implementation of a benchmarking and validation framework for LLM outputs, with clear, measurable quality dimensions and automated checks.
- Design and implement a comprehensive benchmarking framework that measures LLM output quality across multiple dimensions, including:
- Technical correctness, such as valid XHTML and IXBRL schema compliance.
- SEC validation compliance, ensuring generated XBRL instance documents pass SEC validation without errors.
- Completeness, verifying all required facts are tagged (including hidden or non-obvious data points).
- Create complexity “tranches” for benchmarking, ranging from single fund / single share class through multi-fund scenarios and 200+ share class cases, and define success metrics for each tier.
- Build test automation for web and HTML-level testing using Playwright (with no existing automation today), including the first stable automated test harness.
- Create measurable progress indicators that track improvements as the AI tagging core is enhanced and refined.
- Validate and score LLM outputs against both regulatory and business requirements, producing repeatable results suitable for ongoing quality evaluation.
- Collaborate with the R&D team (focused on AI tagging) and production engineering to identify gaps, prioritize iteration steps, and support production readiness.
Deseable
- Experience with scoring models and evaluation dashboards for benchmarking results.
- Previous work on SEC submission tooling, XBRL instance document generation, or schema/validation tooling at scale.
- Experience with financial or compliance testing and validation
- Familiarity with SEC filing requirements and investment company regulations
- Background in data quality validation and metrics design
- Experience with CI/CD pipelines and DevOps practices
Beneficios
- Contrato a largo plazo.
- 100% Remoto.
- Vacaciones y PTOs
- Posibilidad de recibir 2 bonos al año.
- 2 revisiones salariales al año.
- Clases de inglés.
- Equipamiento Apple.
- Plataforma de cursos en linea
- Budget para compra de libros.
- Budget para compra de materiales de trabajo
- mucho mas..