Senior AI Solution Engineer with MS Foundry, MCP

We are seeking a Senior AI Solution Engineer with expertise in MS Foundry and MCP to build automated data ingestion pipelines and data scraping mechanisms on Azure Databricks that fuel the enterprise AI platform and semantic layer. This role focuses on gathering structured and unstructured data, automating workflows via Azure DevOps CI/CD pipelines, and exposing clean data models formatted via Open Knowledge Format directly to Databricks semantic layers, AI Search, and AI agent runtimes.

Responsibilities Build high-throughput data ingestion pipelines and data scraping engines using Python, PySpark, and Databricks Auto Loader to collect unstructured document content, external registry feeds, and enterprise data Structure and expose ingested data within the Databricks semantic layer, organize datasets using Open Knowledge Format standards for seamless consumption by AI Search, Databricks Genie Spaces, and LLM reasoning engines Automate ingestion pipeline deployments, scraping job schedules, and code releases using Azure DevOps and Databricks Asset Bundles (DABs) Scrape and ingest healthcare and insurance data sources, including regulatory databases, public registries, and unstructured medical/claims files, to support AI agent workflows Implement schema enforcement, automated ingestion failure alerts, and job telemetry within Databricks Workflows and Azure DevOps Requirements Extensive experience (5+ years) building high-volume data ingestion pipelines using Azure Databricks, PySpark, and Auto Loader Hands-on experience developing data scraping routines and extraction engines for complex, unstructured data sources Proven ability to structure ingested datasets using the Open Knowledge Format within Databricks semantic layers and AI Search for downstream GenAI consumption Track record of managing Databricks ingestion workflows and deployment pipelines using Azure DevOps and Databricks Asset Bundles (DABs) English proficiency at B2 level or higher Armenia (Prod) We connect like-minded people Delivering innovative solutions to industry leaders, making a global impact Enjoyable working environment, whether it is the vibrant office or the comfort of your home Opportunity to work abroad for up to two months per year Relocation opportunities within our offices in 55+ countries Corporate and social events We invest in your growth Leadership development, career advising, soft skills and well-being programs Certifications, including GCP, Azure and AWS Unlimited access to EPAM's internal learning database Free English classes with certified teachers We cover it all Participation in the Employee Stock Purchase Plan Monetary bonuses for engaging in the referral program Comprehensive medical & family care package Four trust days per year for personal needs Discounts for fitness clubs Benefits package (hotels, restaurants, stores and services) Georgia (Production) We connect like-minded people Delivering innovative solutions to industry leaders, making a global impact Enjoyable working environment, whether it is the vibrant office or the comfort of your own home Opportunity to work abroad for up to two months per year Relocation opportunities within our offices in 55+ countries Corporate and social events We invest in your growth Leadership development, career advising, soft skills and well-being programs Certifications, including GCP, Azure and AWS Unlimited access to EPAM's internal learning database Free English classes with certified teachers We cover it all Participation in the Employee Stock Purchase Plan Monetary bonuses for engaging in the referral program Comprehensive medical & family care package Five trust days per year (sick leave without a medical certificate) Benefits package (sports activities, a variety of stores and services) Kazakhstan (Prod) We connect like-minded people: Delivering innovative solutions to industry leaders, making a global impact Enjoyable working environment, whether it is the vibrant office or the comfort of your own home Opportunity to work abroad for up to two months per year Relocation opportunities within our offices in 55+ countries Corporate and social events We invest in your growth: Leadership development, career advising, soft skills and well-being programs Certifications, including GCP, Azure and AWS Unlimited access to EPAM's internal learning database Free English classes with certified teachers Discounts in local language schools, including online courses for the Kazakh language We cover it all: Participation in the Employee Stock Purchase Plan Monetary bonuses for engaging in the referral program Medical & family care package Six trust days per year (sick leave without a medical certificate) Coverage of psychology sessions of your choice Benefits package (sports activities, a variety of stores and services) Housing support program: preferential mortgage access via Otbasy Bank partnership Kyrgyzstan (Prod) We connect like-minded people: Delivering innovative solutions to industry leaders, making a global impact Enjoyable working environment, whether it is the vibrant office or the comfort of your own home Opportunity to work abroad for up to two months per year Relocation opportunities within our offices in 55+ countries Corporate and social events We invest in your growth: Leadership development, career advising, soft skills and well-being programs Certifications, including GCP, Azure and AWS Unlimited access to EPAM's internal learning database Free English classes with certified teachers We cover it all: Monetary bonuses for engaging in the referral program Medical & family care package Six trust days per year (sick leave without a medical certificate) Coverage of psychology sessions of your choice Discounts for fitness clubs and sports programs Benefits package (sports activities, a variety of stores and services) Uzbekistan (Prod) We connect like-minded people: Delivering innovative solutions to industry leaders, making a global impact Enjoyable working environment, whether it is the vibrant office or the comfort of your own home Opportunity to work abroad for up to two months per year Relocation opportunities within our offices in 55+ countries Corporate and social events We invest in your growth: Leadership development, career advising, soft skills and well-being programs Certifications, including GCP, Azure and AWS Unlimited access to EPAM's internal learning database Free English classes with certified teachers Discounts in local language schools, including offline courses for the Uzbek language We cover it all: Monetary bonuses for engaging in the referral program Medical & family care package Four trust days per year (sick leave without a medical certificate) Discounts for fitness clubs, dance schools and sports programs Benefits package (sports activities, a variety of stores and services)