Senior Data Engineer Big Data & Cloud Integration L3
Select MindsDallas, Texas
Automate your job search with Sonara.
Submit 10x as many applications with less effort than one manual application.1
Reclaim your time by letting our AI handle the grunt work of job searching.
We continuously scan millions of openings to find your top matches.
Job Description
Benefits:
- HYBRID
- Competitive salary
- Opportunity for advancement
Senior Data Engineer – Big Data & Cloud Integration
DALLAS, TX - HYRBID
LONG TERM
IN-PERSON INTERVIEW
Data Engineer - L3 ROLE
- Translate complex cross-functional business requirements and functional specifications into logical program designs and data solutions.
- Partner with the product team to understand business needs and specifications.
- Solve complex architecture, design and business problems.
- Coordinate, Execute and participate in component integration (CIT) scenarios, system integration testing (SIT), and user acceptance testing (UAT) to identify application errors and to ensure quality software deployment.
- Continuously work with cross-functional development teams (Data Analysts and Software Engineers) for creating PySpark jobs using Spark SQL and help them build reports on top of data pipelines.
- Build, test and enhance data curation pipelines, integrate data from a wide variety of sources like DBMS, File systems and APIs for various OKRs and metrics development with high data quality and integrity.
- Execute the development, maintenance, and enhancements of data ingestion solutions of varying complexity levels across various data sources like DBMS, File systems (structured and unstructured), APIs and Streaming on on-prem and cloud infrastructure.
- Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms.
- Work with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas.
- Develop spark applications in PySpark on distributed environment to load huge number CSV files with different schema in to Hive ORC tables.
- Perform ETL transformations on the data loaded into Spark Data Frames and do the in-memory computation.
- Develop and implement data pipelines using AWS services such as Kinesis, S3 to process data in real-time.
- Work with monitoring, logging and cost management tools that integrate with AWS.
- Schedule the spark jobs using Airflow scheduler to monitor their performance.
....
- Partner with the product team to understand business needs and specifications.
- Solve complex architecture, design and business problems.
- Coordinate, Execute and participate in component integration (CIT) scenarios, system integration testing (SIT), and user acceptance testing (UAT) to identify application errors and to ensure quality software deployment.
- Continuously work with cross-functional development teams (Data Analysts and Software Engineers) for creating PySpark jobs using Spark SQL and help them build reports on top of data pipelines.
- Build, test and enhance data curation pipelines, integrate data from a wide variety of sources like DBMS, File systems and APIs for various OKRs and metrics development with high data quality and integrity.
- Execute the development, maintenance, and enhancements of data ingestion solutions of varying complexity levels across various data sources like DBMS, File systems (structured and unstructured), APIs and Streaming on on-prem and cloud infrastructure.
- Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms.
- Work with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas.
- Develop spark applications in PySpark on distributed environment to load huge number CSV files with different schema in to Hive ORC tables.
- Perform ETL transformations on the data loaded into Spark Data Frames and do the in-memory computation.
- Develop and implement data pipelines using AWS services such as Kinesis, S3 to process data in real-time.
- Work with monitoring, logging and cost management tools that integrate with AWS.
- Schedule the spark jobs using Airflow scheduler to monitor their performance.
....
Flexible work from home options available.
Compensation: $60.00 - $65.00 per hour
About Us
We work to deliver profitability in your business – with effective communication, consulting, and interactive solutions. Following an Agile Work Approach, we make sure you get the ideal solutions at minimum expenses.
Work Approach
Our Philosophy
Our Philosophy starts-and-ends at the Client-first approach. Be it understanding your business requirements to choosing the right technologies, we work as a collective team that takes all the possible steps to grow continuously towards our common goal.
Our Philosophy starts-and-ends at the Client-first approach. Be it understanding your business requirements to choosing the right technologies, we work as a collective team that takes all the possible steps to grow continuously towards our common goal.
Work Policy
We promote a collaborative work environment. We involve everyone working in the organization in community decisions and encourage them to think from a broader perspective. Our work process promotes flexibility and we maintain a high level of discipline at different levels of execution.
The Future
SelectMinds have years of experience in the domain helps us understand the need-of-the-hour better. This understanding drives us to a better future with every minute ticking. We believe we will be taking off major businesses from their flagship positions, with the products we are eyeing today.
Automate your job search with Sonara.
Submit 10x as many applications with less effort than one manual application.