Senior Data Engineer

Job Description

JOB INFORMATION

Title:

Senior Data Engineer

Officer Title:

Grade:

29

FLSA Classification:

Exempt

Date Last Edited:

5/14/2025

Reports To:

Manager Data Platform

Department:

IT

URL Link

https://hvfcu.jdxpert.com/ShowJob.aspx?EntityID=2&jobcode=1123

PRIMARY FUNCTION

Design, build, and optimize our data infrastructure, leveraging the power and scalability of Databricks within our Azure environment. Maintain a deep expertise in Databricks and its ecosystem in order to build high-performance data pipelines, ensure data quality, and enable advanced analytics and machine learning initiatives.

•

Design, develop, and maintain robust and scalable data pipelines primarily using Databricks, leveraging Spark (PySpark or Scala), Delta Lake, and Databricks Workflows.

•

Build and optimize data lakehouse solutions on Azure Data Lake Storage Gen2 integrated with Databricks

•

Implement and enforce data quality frameworks and data governance policies within Databricks using tools and features like Delta Lake's ACID properties and data validation techniques

•

Identify and resolve performance bottlenecks in Databricks-based data pipelines and Spark applications

•

Collaborate closely with analysts to understand their data requirements for advanced analytics, providing them with curated and performant datasets within Databricks

•

Automate data ingestion, transformation, and deployment processes, potentially integrating with CI/CD tools and Databricks MLOps

•

Provide technical leadership and mentorship to junior data engineers on Databricks best practices and performance tuning

•

Document Databricks notebooks, data pipelines, data models, and configurations

•

Ensure data security and compliance within the Databricks environment, adhering to relevant security standards and best practices

•

Troubleshoot and resolve complex data-related issues within the Databricks ecosystem

•

Work independently and contribute to immediate and cross-functional teams

•

Participate in design discussions

•

Analyze problem statements, break down problems, and provide solutions

•

Follow an agile-based approach to deliver data products

•

Take ownership of large tasks, delivering on time, and mentoring team members

•

Oversee the maintenance and enforcement of SDLC and ITSM processes, change management procedures, departmental standards, style requirements and all other procedures necessary for data and system integrity

•

Adhere to all Credit Union policies, procedures, and regulatory agency requirements. Participate in all required and recommended training and development including, but not limited to, Bank Secrecy Act training (BSA) and demonstrate attained knowledge. Participate on Credit Union teams, projects and strategic initiatives when the opportunity arises. Perform additional duties and special projects as assigned.

•

Embrace and apply HVCU’s guiding principles to all activities and responsibilities. This includes the Credit Union’s Mission, Vision, Core Values, Employee and Member Value Propositions, Sales and Service Model, and commitment to Lean Six Sigma practices. Support the Credit Union’s initiatives by demonstrating teamwork and professionalism.

•

Responsible for regular and predictable attendance including punctuality.

Bachelor’s Degree or at least 2 years equivalent experience for each undergraduate year not completed; undergraduate degree cannot substitute for minimum number of years of experience.

Computer Science, or related field

X

Master’s Degree

X

Microsoft Certified in Azure Data Engineer

X

Databricks Certified in Data Engineer Associate

X

Databricks Certified in Data Engineer Professional

X

Minimum 5 Years

Data engineer

X

Minimum 5 Years

Data Warehousing, Data Lakes, and big data platforms

X

Minimum 5 Years

Python

X

Minimum 3 Years

Databricks

X

•

Strong proficiency in SQL and experience with Microsoft SQL Server databases.

•

Advanced proficiency in Microsoft Office Software (Word, Excel, Outlook, PowerPoint, etc.)

•

Strategic thinker with the ability to apply a business mindset to data issues and initiatives, crafting cross-business strategies and plans for multiple stakeholders.

•

Advanced verbal, written and presentation skills.

•

Ability to work in a fast-paced environment and deal with a variety of internal/external customer and vendor situations.

PHYSICAL DEMANDS

Frequency

Physical Demand

Weight (lbs.)

Frequently

Sit for prolonged periods of time

Frequently

Required to communicate verbally

Frequently

Normal range of vision and hearing abilities required

Occasionally

Lift and/or move objects

30

Occasionally

Travel outside of work location to attend meetings and training programs

Frequently

Required to skillfully operate a computer, telephone and other standard office equipment

WORK ENVIRONMENT

•

Typically the noise level in the work environment is low to moderate.

•

May experience occasional job stress in dealing with difficult situations and/or high volume.

•

High energy, high impact atmosphere.

•

There are no significant hazardous conditions.

View Competencies

Leadership

Knowledge of, and ability to use strategies and skills to enlist others in setting, embracing and achieving objectives.

Level 1

1. Explains several leadership theories.

2. Lists 10 characteristics (charisma, integrity, etc.) that distinguish leaders from non-leaders.

3. Provides examples of how leaders signal their goals, intentions, and attitudes through behavior.

4. Describes how a leader's mood affects group behavior and productivity.

Service Excellence

Consistently incorporates Hudson Valley's member and employee value proposition, reinforcing organizational culture and brand differentiators.

Level 2

1. Creates a positive team environment by celebrating successes and driving excellence.  2. Dedicates required time and energy to assignments or tasks to ensure that no aspect of the work is neglected; works to overcome obstacles to completing tasks or assignments.  3. Effectively communicates the connection between culture and the business strategy.  4. Implements effective strategies and activities that promote long term customer loyalty.  5. Inspires and strives for excellence in all aspects of work including setting high performance goals for oneself and others.

Results Driven

Drives critical activities to completion. Meets organizational goals and customer expectations.

Level 1

1. Clearly communicates expectations and what results look like. Uses ongoing communication and feedback to promote team motivation to achieve objectives.  2. Complies with specific policies or practices based on laws or regulations governing these practices.  3. Effectively navigates and utilizes formal and informal communication and decision-making channels.  4. Gives honest, timely feedback, including difficult discussions about taking ownership for one's decisions, actions and results.  5. Identifies barriers to goal achievement and provides necessary tools, resources, and development to overcome obstacles.

Resiliency

Adapts positively to changing environments in order to achieve business results.

Level 1

1. Celebrates incremental success in creating change and assists others in seeing the role they can play in transforming the organization.  2. Leverages intellectual curiosity to remain current and prepare for potential change.  3. Open to change and new information; rapidly adapts to new information, changing conditions, or unexpected obstacles.   4. Perceives, assesses and helps manage the emotions of self and others while simultaneously exhibiting drive and commitment to results.  5. Recovers quickly from setbacks and maintains composure under stress.

Data Networks

Knowledge of and the ability to manage an organization's data networks for internal and external information transmission.

Level 3

1. Designs, implements and supports multiple types of data networks.

2. Develops alternative scenarios for network applications; explains network inter-connectivity.

3. Designs optimal routes between any two points on a network.

4. Compares and contrasts major features of alternative data network technologies.

5. Evaluates the effectiveness of network topology, physical and logical networks, links and interfaces.

6. Analyzes the advantages and drawbacks of cellular, satellite and microwave technologies.

This job description is not designed to cover or contain a comprehensive listing of activities, duties, or responsibilities that are required of the employee in this position.  Additional functions and requirements may be assigned by supervisors as deemed appropriate, with or without notice.

Nothing contained in this material creates a contract of employment between an employee and Hudson Valley Credit Union or otherwise alters the at-will nature of the employment relationship.

Are you able to perform the essential functions listed in this job description with or without reasonable accommodation?  YES_______  NO_______

By signing below you certify that you have received this job description and understand the requirements, essential functions and duties of the position.  If you are currently employed in this position, your signature acknowledge that this job description accurately describes the position, including its essential functions.

Name:  _____________________________________________

Signature: _____________________________________________

Date:  _____________________________________________ 

Please also note that Hudson Valley Credit Union respects the religious beliefs and practices of employees and will, on request, reasonably accommodate its employees when a reasonable accommodation is available that does not create an undue hardship on the Credit Union’s business.  Therefore, unavailability for work due to religious observances or practices may not be a bar to employment.