Lead Data Engineer

POSITION SUMMARY

Salary - $149,251.46

Responsible for leading development and maintenance of data architectures, including back-end infrastructures, complex data integrations and associated processes. Drives department's long-term data strategy, by spear-heading future advancements and incorporating industry standards. Owns the overall design, evaluation, selection, implementation and success of data processes and downstream applications. Leads data engineering team, including project management and technical leadership. Monitors data eco-system for the reliability, quality and usefulness of agency data as it relates to the needs of enterprise Data Scientists, Data Engineers, Analysts, related stakeholders and departmental direction.

PRIMARY RESPONSIBILITIES

• Develops, constructs, tests and maintains data architecture, including databases, data processing systems and related applications.
• Determines requirements that will be used to develop and evolve data models and algorithms.
• Develops custom applications and infrastructure as necessary.
• Uses advanced analytics methods to extract, transform, and load business data.
• Researches and deploys data infrastructure best practices. Builds and optimizes data pipelines to support cross-functional analytics, automation and programming needs.
• Supervises data engineering team in project management, building data pipelines and quality assurance to ensure consistent and reliable data for analysis.
• Monitors progress towards project objectives and checks teams' work for accuracy. Ensures all targets and requirements are met and delivered on time.
• Oversees and advises on appropriate use of data warehousing techniques and coding best practices.
• Spearheads short- and long-term direction of department data eco-system and strategies.
• Liaises with other departments (Data Analytics, Technology, Innovation, etc.), and end user audiences (executive office, field department VPs, etc.), to build reliable data infrastructure conducive to cross-departmental maintenance and use.
• Leads new efforts to acquire and centralize data from disparate systems.
• Drives department's long-term data engineering strategy.
• Hires, trains, develops, monitors and evaluates staff. Reviews and recommends personnel actions for approval.
• Performs related duties as assigned.

MANAGEMENT RESPONSIBILITIES

Reporting to this position are the following jobs:

• Data Scientist (Indirect)
• Senior Analyst, Performance Management (Indirect)
• Analyst, Performance Management (Indirect)
• Senior Data Engineer (Direct)
• Data Engineer (Direct)

CHALLENGES

• Working with and managing massive amounts of data sourced from several disparate data systems and applications.
• Implementing effective time and project management processes to deliver data services on time and on budget.
• Balancing multiple high-priority requests simultaneously while adapting to rapidly changing demands and meeting project deadlines.

EDUCATION/EXPERIENCE REQUIREMENTS

Bachelor's degree in Statistics, Mathematics, Computer Science, Engineering, or a related field, plus at least six (6) years of progressive experience in data science, analytics, or performance management, including two (2) years of team leadership or mentoring experience, or an equivalent combination of education and experience.

PHYSICAL REQUIREMENTS

• Requires sitting for extended periods of time, standing, visual acumen, manual dexterity, and fingering for working with computer keyboards.
• Requires remaining in a stationary position for extended periods of time and constantly operating a computer and other office productivity machinery.

KNOWLEDGE, SKILLS, AND ABILITIES

• Advanced knowledge of various data programming languages and frameworks (Python, SQL, Java, JavaScript, PHP, C#/.NET, VueJS, etc.).
• Advanced experience in custom and/or industry-standard data migration, integration, and ETL tools and techniques.
• Advanced knowledge of big data frameworks and tools (Hadoop, Spark, MongoDB, Cassandra, etc.)
• Advanced knowledge of programming development requirements and use of source/revision control systems (Git, Mercurial, etc.)
• Advanced knowledge of cloud-based data infrastructure (AWS, Azure, Google, Vertica)
• Advanced knowledge of a range of RDBMS platforms (Postgres, Redshift, DynamoDB, Oracle, SQL Server, MySQL, DB2, etc.)
• Advanced knowledge of specialized database architectures and computing environments.
• Advanced knowledge of continuous delivery and deployment pipelines.
• Advanced analytical, problem-solving, and decision-making skills.
• Advanced report preparation and presenting skills.
• Advanced oral and written communication skills to communicate recommendations to enable decision-making.
• Advanced collaboration skills with cross-functional stakeholders to understand the business usage of data.
• Working knowledge of artificial intelligence (AI) and machine learning.
• Advanced interpersonal and team skills across a variety of fields and management levels.
• Advanced project management skills.
• Advanced organization and time management skills.
• Strong ability to effectively analyze and translate data engineering challenges to the business.
• Strong ability to manage large amounts of data and attention to detail.
• Strong ability to multitask competing projects and deadlines for completion.

WORKING CONDITIONS

• General office environment.
• Required to travel to various CTA field locations and transit property locations.
• Subject to typical field conditions such as noise, dust, grease, and moving vehicles when on-site.

EQUIPMENT, TOOLS, AND MATERIALS UTILIZED

• Standard office equipment.
• Personal computer and related software (MS Word, PowerPoint, Excel, Power BI).