What are the responsibilities and job description for the Data Platform Engineer position at Varite, Inc?
Pay Rate: $77.46 - $79.57
Duties:
Cloud Infrastructure Administration and Automation:
Duties:
- Setup and maintain production scale Databricks environment on public cloud such as Microsoft Azure and AWS (Amazon Web Services)
- Setup and maintain production scale data storage such as ADLS (Azure Data Lake Storage) and AWS S3 for multiple tenant teams using our Data Platform
- Setup and maintain production scale micro services to support the daily operation of our data platform. Services include job scheduling, security, financial, and administrative services, etc.
- Provide triage and guidance to the team on various support issues raised by our tenants
- Develop tools and automation solutions for configuration management, service deployments, monitoring, and alerting to assist with daily RTB (Running the Business) operations
- Budget and monitor cloud spend, always think of ways to avoid cloud resource wastage, utilize 3rd party tools, or develop your own tools to help the team with cost optimization
- Assure security and privacy compliance and implement Adobe Security & Compliance solutions to lock down data stored in our data lake
- Explore GenAI technologies and find opportunity to integrate them with our data platform, providing platform enhancement or improving platform user experience in the end
- Work with various 3rd party vendors for troubleshooting, proof of concept, and other collaborative projects to enhance our product.
Cloud Infrastructure Administration and Automation:
- AWS
- Azure
- ADLS Gen2
- AWS S3
- Hive or MySQL
- MongoDB
- Vector Databases
- Databricks Workspace, includes but not limited to
- Unity Catalog
- Vector Search
- SQL Warehouse
- Serverless Compute
- Spark workloads
- Airflow and DAGs
- Azure Kubernetes Service or Elastic Kubernetes Service
- Collibra
- Neo4J
- Metric Insights
- Databricks System Tables
- Prometheus
- Splunk
- ELK
- Power BI
- Servers with Linux system
- Kubernetes environment
- Jira
- Service Now
- BS in Computer Science, Computer Engineering, or similar.
Salary : $77 - $80
