Lead Software Development Engineer

Through NRI (formally Core BTS) Resource Management Services (RMS), we offer custom talent solutions to help our clients meet their evolving technology and business needs. We help effectively match the right technology professional to their organization, recruiting for contract, contract-to-hire, and direct roles. This is not a position with Core BTS d/b/a NRI but at a client site of our organization.

Position Summary

As a Lead Data Engineer, you will serve as a key architect in evolving our data infrastructure, under the technical direction of the Director of Data Engineering. Your primary responsibilities will include leading the design, implementation, and continuous enhancement of our data architectures, data pipelines, and interfaces to maximize scalability and efficiency. With advanced proficiency in SQL, Python, and cloud technologies (Azure, AWS, or Google Cloud), you will spearhead the development of data solutions that are synchronized with our strategic objectives. Your deep understanding of Big Data concepts, including Spark and Cloud ETL tools like Databricks, will be crucial in addressing complex data challenges. Leveraging Agile/SCRUM methodologies, you will guide innovative and prompt project completions. This hands-on leadership role involves mentoring senior data engineers and fostering a culture of excellence and proactive growth. As a leading figure in our team, you will collaborate extensively with data scientists, BI teams, software engineers, and business stakeholders to devise and execute effective data engineering strategies that drive significant business impact.

Primary Duties, Responsibilities, Or Activities Required By Job

• Strategic Data Pipeline Design & Optimization: Lead the architectural design and optimization of robust, scalable data pipelines using SQL, Python, and cloud-based ETL tools such as Databricks. Oversee data flow and processing to support large-scale operations and ensure system-wide efficiency.
• Advanced Data Modeling: Direct the development and refinement of complex data models to accurately mirror critical business processes, ensuring seamless integration with our comprehensive data architecture, including Big Data frameworks like Spark.
• Data Architecture Leadership: Shape and enhance our data architecture strategy, making high-level decisions on data storage, consumption, integration, and management across cloud platforms (Azure, AWS, or Google Cloud).
• Agile/SCRUM Leadership: Orchestrate Agile/SCRUM frameworks to drive efficient project delivery. Lead sprints and stand-ups, implementing these methodologies to optimize development processes across teams.
• Senior-Level Collaboration: Lead collaboration efforts with data scientists, BI teams, and engineering groups to translate intricate data requirements into executable engineering strategies. Act as a primary technical liaison among stakeholders.
• Mentorship of Senior Engineers: Mentor senior data engineers and technical team leads, instilling best practices in SQL, Python, and cloud technologies, while nurturing a culture of excellence and continuous advancement.
• Quality & Governance Oversight: Establish and enforce the highest data quality standards and governance policies, ensuring reliability and regulatory compliance across all data operations.
• Performance Tuning Expertise: Continuously monitor and refine the performance of data infrastructure, identifying and resolving high-level bottlenecks or inefficiencies in cloud and Big Data environments.
• Innovation and Strategic Implementation: Stay at the forefront of emerging data engineering technologies and methodologies, leading the evaluation and strategic implementation of innovative tools and practices that enhance our capabilities.
• Documentation and Knowledge Dissemination: Oversee the creation of detailed documentation for data processes, pipelines, and architectures, ensuring clarity, consistency, and easy maintenance across technical teams.
  
  

Other Secondary Functions

• Responsible for sharing on-call rotation and off-hours outage escalations support with colleagues.
• Provide support for existing legacy data solutions and develop migration paths to new platforms as required / necessary.
  
  

Education

Required Minimum Education & Experience

• Bachelor’s Degree in Computer Science, Data Science, Information Technology or other quantitative disciplines such as Science, Statistics, Economics, or Mathematics.
• Bachelor’s Degree in Computer Science, Data Science, Information Technology, or other quantitative disciplines such as Science, Statistics, Economics, or Mathematics.
  
  

Essential Experience

• 10 years of progressive experience in data engineering, with proven expertise in designing, implementing, and leading initiatives for optimizing databases and data pipelines.
• Extensive hands-on experience with SQL Server, Oracle, or other relational database management systems (RDBMS).
• Advanced proficiency in SQL and Python for sophisticated data manipulation and analytics.
• Demonstrated leadership in data modeling and architecture for both analytics and transactional systems within large-scale environments.
• At least 2 years of experience leading teams of data engineers: Proven ability to mentor and guide teams in the development and optimization of data systems, fostering a collaborative and innovative working environment.
• Strong hands-on experience with Mobile Development
  
  

Cloud And Big Data Experience

• Expertise with at least one major cloud data platform (Azure, AWS, Google Cloud) with extensive application in data engineering projects.
• Deep knowledge of Big Data technologies such as Spark and Cloud ETL tools like Databricks, with a focus on scalability and real-time processing capabilities.
  
  

Methodology And Tools

• Senior-level experience with Agile and SCRUM methodologies, leading successful project delivery in a dynamic development environment.
• Skilled in developing data models for integration and analysis that support complex business intelligence and data analytics initiatives.
  
  

Preferred Qualifications

• Education: Master’s Degree in Computer Science, Data Science, Information Technology or other quantitative disciplines such as Science, Statistics, Economics, or Mathematics.
• Experience with the Microsoft Azure technology stack or similar technologies in competing platforms.
• Practical knowledge of data analytics and visualization tools to aid in data-driven decision making and reporting.
  
  

Certifications

• Professional certification in Agile and SCRUM methodologies (e.g., Certified ScrumMaster (CSM), SAFe Agilist).
• Certifications in Python and SQL programming (e.g., Microsoft Certified: Python Programming Specialist, Oracle SQL Certification).
• Certifications in cloud services relevant to the job (e.g., AWS Certified Solutions Architect, Google Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate).
• Big Data certifications (e.g., Cloudera Certified Professional (CCP): Data Engineer, Databricks Certified Professional Data Scientist).
  
  

Willing to attain certification, if not currently certified.

Required Knowledge, Skills & Abilities (KSA’s)

Knowledge

• Fluent Communication: Ability to articulate complex data concepts and project updates clearly to both technical and non-technical stakeholders.
• Strong Data Analysis Ability: Expertise in analyzing large datasets to derive insights and inform business decisions.
• Proficiency in SQL and Python: High level of skill in SQL and Python for data analysis, data manipulation and scripting.
• ETL/ELT Architecture: In-depth knowledge of developing and managing ETL and ELT architectures using various tools and frameworks.
• Cloud Experience: Experience with cloud platforms such as Azure, AWS, or Google Cloud, and their respective data services and tools.
• Big Data Concepts: Understanding of Big Data technologies and frameworks, including Spark and Cloud ETL tools such as Databricks.
• Agile and SCRUM Knowledge: Familiarity with Agile methodologies and SCRUM practices, capable of integrating these into project management and daily workflows.
• Quality Assurance and Data Governance: Knowledge of data quality standards and governance, ensuring data integrity and compliance across all processes.
  
  

Skills

• Collaboration: Ability to work effectively with cross-functional teams, including data scientists, BI analysts, and software engineers, to implement data solutions.
• Mentorship and Leadership: Skills in mentoring junior engineers and leading project teams to promote knowledge sharing and professional growth within the team.
• Innovation and Continuous Learning: Commitment to staying updated on the latest industry trends and technologies in data engineering and implementing them as relevant.
  
  

Ability

• Ability to interact with customers one-on-one or in large groups
• Ability to work independently with remote supervision.
• Ability to build in receiving feedback as part of the development process and seek consistent and constructive feedback.
• Ability to embrace accountability and ownership.