Overview

Senior Quality Assurance Engineer Jobs in Montreal, Quebec, Canada at Millennium Software and Staffing Inc

Title: Senior Quality Assurance Engineer

Company: Millennium Software and Staffing Inc

Location: Montreal, Quebec, Canada

Hello ,

We have the following openings at Montreal Canada 4 days a week on Site. Kindly send your updated resume if interested at [email protected] and you may reach out to me at 248-233-6240.

Job Title

QA Engineer(AI)

Location

Montreal Canada 4 days a week on Site

Duration

12 months and possible Contract-To-Hire any time after 90 days

Job Description: CGEMJP00346687 – QA Engineer(AI) – Montreal Canada 4 days a week on Site

4 days a week on site, F2F interview required

JOB DESCRIPTION:

Job Details:

Must have Skills:

  • AI LLM Testing Expertise prompt engineering hallucination detection bias safety evaluation.
  • RAG Vector DB Testing retrieval accuracy embeddings LangChain, LangGraph.
  • Automation Coding, Python, Java API testing CI/CD test frameworks.

Key Responsibilities:

AI Systems and Models Testing:

  • Design and execute comprehensive test strategies for AI systems and models including prompt engineering output evaluation and bias safety testing Develop deep understanding of LLM behavior tokenization embeddings attention mechanisms and inference to anticipate failure modes Construct effective prompts recognize hallucinations and off target outputs and assess quality across accuracy tone coherence and bias dimensions Apply evaluation metrics specific to generative AI and establish appropriate thresholds
  • Test AI systems integrated with RAG pipelines and knowledge bases validating data quality and retrieval accuracy as they impact model outputs Understand vector database mechanics similarity search thresholds embedding drift and test edge cases including near duplicate documents sparse vs dense embeddings and performance under scale Leverage LangChain and LangGraph frameworks to read code understand chain and graph construction identify failure points and write test harnesses Validate integration points using MCPs testing tool availability and error handling

Test Strategy and Planning:

  • Define and execute comprehensive test strategies for securitization platforms ensuring coverage across functional regression integration and performance testing.
  • Establish testing standards and best practices that span both traditional QA and AI specific validation.

Test Automation and Framework Development:

  • Design build and maintain automated test suites to accelerate release cycles and improve coverage Leverage AI and ML tools to enhance test coverage improve efficiency and reduce regression cycles.
  • Securitization Lifecycle QA Validate end-to-end deal workflows including setup structuring processing and distributions.
  • Ensure data integrity across upstream and downstream systems through reconciliation testing and reporting.

Release and Regression Testing:

  • Coordinate regression testing for platform releases patches and infrastructure changes Ensure stability and backward compatibility particularly during critical processing windows

Cross Functional Collaboration:

  • Partner with developers business analysts and product owners to clarify requirements identify edge cases and ensure testability of new features.
  • Lead defect triage sessions prioritize issues based on business impact and maintain clear documentation through to closure.

Quality Leadership and Reporting:

  • Define and monitor key quality indicators including defect density test coverage and automation rates.
  • Present findings to leadership and recommend improvements.
  • Mentor junior QA team members and foster a culture of quality across the team.
  • Production Support Provide production support during critical processing windows investigate incidents and coordinate root cause analysis and remediation efforts.

Qualifications:

Experience:

  • 7 years of quality assurance or quality engineering experience with at least 3 years in a lead or senior capacity.
  • Strong domain knowledge in securitization capital markets or similar asset classes.

Technical Skills:

  • Handson experience with test automation tools Selenium, Robot Framework, Playwright or similar.
  • Proficiency in programming languages including Java and Python with demonstrated framework implementation expertise.
  • Handson API automation and backend system validation experience.
  • Proficiency in database query development data validation and reconciliation testing.
  • Experience with CI/CD pipelines and DevOps practices Jenkins, GitHub or similar

AI and ML Competencies:

  • Core understanding of LLM architecture and behavior.
  • Handson experience with LangChain and or LangGraph frameworks.
  • Knowledge of RAG pipelines, vector databases and agentic solutions.
  • Familiarity with Model Context Protocols MCPs and integration testing.
  • Understanding of bias safety and red team testing methodologies for AI systems

Soft Skills and Mindset:

  • Demonstrated passion for automation continuous learning and solving open ended problems.
  • Excellent analytical problem solving and communication skills.
  • Ability to manage multiple priorities in a fast-paced deadline driven environment.
  • Ability to mentor and lead teams while maintaining focus on quality excellence

Regards,

Manish Jakhar

Millennium Software Inc.

Cell: 248-233-6240

2000 Town Center, Suite 300, Southfield, MI, 48075

Visit us at www.webmsi.com

Apply

Upload your CV/resume or any other relevant file. Max. file size: 800 MB.