We seek a DevOps Engineer to play a pivotal role in maturing our data platform and developer experience. You will be responsible for evolving our DevOps platform, processes and systems with the ultimate aim of enhancing developer velocity and practices, improving code and data quality, and maintaining a well-managed data platform.
Key Responsibilities:
SDLC
Key Responsibilities:
SDLC
- Databricks Asset Bundles (DABs): Enhance our DABs processes to enhance developer experience and velocity.
- CI/CD Pipeline Development: Build end-to-end automation in Azure DevOps to test and promote code across our environments (covering ADF, ADB and PowerBI)
- Infrastructure as Code (IaC): Automate the provisioning of Databricks workspaces clusters, and projects using Terraform or relevant templates.
- Governance: Mature and manage security policies and access controls
- ADF Management: Manage our ADF resources, optimizing for reliability and cost.
- Monitoring & Cost Optimization: Enhance observability across storage and compute to ensure cost effective operations
- Bachelor’s degree in a relevant field (e.g., Data Engineering, Software Engineering, Computer Science)
- Extensive hands-on experience with Azure Cloud services; strong understanding of resource provisioning, networking, and security in Azure.
- Deep experience with Azure Databricks administration (clusters, scaling, job scheduling, monitoring).
- Strong experience with IaC (Terraform or Bicep), managing changes via code, and infrastructure versioning.
- Competence with version control (Git) and implementing CI/CD pipelines for data workloads/infra.
- Familiarity with Spark internals (how compute works, resource allocation, partitioning, memory), to aid in optimizing infrastructure.
- Knowledge of cost management on cloud (spot/preemptible VMs, auto-scaling, rightsizing).
- Strong scripting/automation skills (Python / PowerShell / Bash).
- Experience working with data engineering, analytics and BI teams; understanding of pipeline flows, data dependencies, and common data bottlenecks.
- Good to have Databricks or Azure certifications