Data science is a fast-growing field; so fast, in fact, that there have been more open positions in data engineering than available engineers for several years. According to data engineer Carlin Eng, companies hiring data engineers have set aggressive hiring goals to keep pace: "Most were looking to double their engineering headcount by the end of the year, and more than double the size of their data engineering teams. More often than not, when I asked engineering leaders about their biggest challenges, hiring was number 1 on the list."

The gap between the number of qualified data engineers and the number of available positions is starting to close as more people choose careers in data science. Even so, there's still a considerable need for engineers to design, build, and maintain the mechanisms for collecting and validating data. Data analysts and data scientists need clean data sets to produce the research that drives modern business strategy, medical research, national security, and many other endeavors. Data engineers build the structures that generate those data sets. In so doing, they construct the foundation on which the entirety of the data science field rests.

Most data engineers are curious and helpful, skilled problem solvers, and obsessed with data. If that sounds like you, keep reading to find out whether your future lies in data engineering. In this guide to how to become a data engineer, we'll cover:

• What is a data engineer?
• Data science versus data engineering
• Kinds of data engineer careers
• Educational commitment to become a data engineer
• Further accreditation or education for a data engineer
• Typical advancement path for a data engineer
• Pros and cons of becoming a data engineer
• Should I become a data engineer?

What is a data engineer?

A data engineer is a professional who creates reliable architectures and interfaces designed to collect a large amount of data from different sources and transform it into a usable format for analysis. That might sound straightforward, but it involves designing the infrastructure (from databases to processing systems) that underpins just about everything that happens in the data science world. Data engineers use all kinds of scripting languages and tools to build and improve upon data analytics systems. What they don't do, however, is much analysis or modeling.

When you become a data engineer, you'll spend your days:

• Extracting data from various sources
• Preparing data as part of ETL (extract, transform, and load) processes
• Evaluating, parsing, and cleaning data sets
• Building complex data pipelines
• Writing ETL logic
• Stitching data together
• Putting code into production
• Working with a database administrator to create data stores
• Exposing those stores to analytical applications
• Using frameworks to serve data

To succeed in this role, you need a solid grasp of systems architecture, programming, database design and configuration, and interface configuration. You need to be every bit as clever and technically skilled as other data science professionals, but you have to be ready to accept the fact that you won't get nearly as much of the glory.

Data science versus data engineering

Data engineering is an essential part of data science; there's actually a substantial overlap between what data engineers do and what data scientists do. Both of these professionals deal with data, and both must be skilled programmers. Both have a crucial part to play in using data to meet organizational goals.

The most significant difference is that data scientists (and advanced analysts) use their skills to interpret data and deliver insights related to it; data engineers use their skills to build the high-performance infrastructure necessary to generate data and ready that data to be interpreted. You could say that data scientists, analysts, and engineers are all members of the same team playing complementary, equally important roles.

Kinds of data engineer careers

Data engineers answer to many different titles... Hadoop developer, ETL developer, BI developer, technical architect, data warehouse engineer, data science software engineer, and quantitative data engineer, to name just a few. They also have different levels of programming experience, though this isn't always reflected in their titles.

These days, the terms "data engineer" and "big data engineer" are often used interchangeably—because increasingly, all data is Big Data—though some people differentiate between the two. Where those people draw the line differs, however. Some say that big data engineers are more focused on open source distributed platforms such as Hadoop, while traditional data engineers are primarily focused on delivering data pipelines. Check listings on sites like Indeed to see how different employers define the role.

Educational commitment to become a data engineer

If you want to join the ranks of data engineers, what you know will be a lot more important than what degree you get—or even whether you get one at all. There are very few data engineering degrees at the undergraduate or graduate levels in the US; you'll find more if you have the resources and the qualifications necessary to study in Europe. Northeastern University, Stevens Institute of Technology, and the University of Wisconsin - Madison offer some of the only master's programs focused specifically on data engineering.

Instead of looking for degrees in data engineering, look for computer science degrees, information systems degrees, data science degrees, big data degrees and analytics degrees that give students the option of choosing a data engineering concentration.

Undergraduate degrees

• Kent State University at Kent offers a Bachelor of Science in Computer Science program with a data engineering concentration
• Pennsylvania State University - Main Campus has a Bachelor of Science in Data Sciences program with a computational data sciences concentration
• Wellesley College offers a Bachelor of Data Science with a dual concentration in computer science and data engineering

Master's degrees

• George Mason University offers a Master of Science in Data Analytics Engineering
• Northwestern College offers a Master of Science in Data Science with a data engineering concentration
• Regis University offers a Master of Science in Data Science with a data engineering specialization

The name of your degree will matter less than the content of the program. Look for programs that have core courses or electives focused on:

• Relational and non-relational database theory and practice
• Data modeling techniques
• Programming
• ETL design
• Database clustering techniques
• Architectural projections

Don't expect to learn everything you'll need to know to become a data engineer in school, however.

Further accreditation or education for a data engineer

Succeeding as a data engineer is all about having the relevant technical skills. Continuing education for data engineers often involves learning to use whatever high-tech tools and programming languages weren't covered in a degree program. Companies hiring data engineers usually ask for experience with:

• Hadoop/Hive
• Java/Scala
• Spark
• Kafka
• SQL and NoSQL
• Python
• Cloud platforms like AWS
• Algorithms and data structures
• Distributed systems
• Tableau
• ElasticSearch
• Data warehousing and ETL tools
• Machine learning
• UNIX, Linux, and Solaris

Unless you decide to pursue a data engineering degree, chances are that you won't find a bachelor's degree program or even a master's degree program that will cover everything you need to know to become a data engineer. The good news is that you can get the skills and knowledge you'll need via online courses on sites like Udemy. These courses will guide you as you learn relevant programming languages and gain hands-on experience using the most common data engineering tools.

There are also certifications for data engineers, though not many. They are usually tool-specific, such as:

• Google's Cloud Data Engineer Certification
• Cloudera's Certified Professional Data Engineer credential
• Microsoft's Certified Solutions Associate in Data Engineering with Azure credential
• Microsoft's MCSE: Data Management and Analytics credential
• IBM's Certified Data Engineer credential

Just be careful not to invest too much money or time in the wrong courses or certifications. In a blog post by data engineer Jesse Anderson about what he looks for when hiring data engineers, he cautions aspiring engineers against taking low-cost online courses and pursuing every certification under the sun. "They're too general, taught by people with not enough knowledge, and they won't help you get a job… You're better off putting your time and money into a personal project that shows true mastery… You have to both internalize the knowledge and practice it. If you've learned passively but never practiced, you won't be able to code a project, and that will come out in an interview. Practice, practice, practice!"

Typical advancement path for a data engineer

Data engineer is not usually an entry-level role. Most employers prefer candidates who have significant experience in coding and working with data.

If you're thinking the best way to advance to this role is to become an analyst first, think again. Even though many data analysts go on to become data scientists, very few make the transition to data engineering. Most data engineers start out as software engineers: this job is all about building tools, frameworks, and infrastructure from the ground up.

Whether you transition into data engineering or you look for jobs right out of school, you will probably follow this advancement path:

• Junior data engineer
• Data engineer
• Senior data engineer
• Lead data engineer
• Head of data engineering
• Chief data officer

If you don't have any software engineering or analytics experience but you want to land a position in data engineering, follow Anderson's advice above: work on one or more projects that showcase what you can do.

Pros and cons of becoming a data engineer

The biggest pro is probably that this job pays well. While a data engineer salary can fall anywhere between $64,000 and $132,000, you'll probably make around $91,000 when you become a data engineer.

Another big pro is that data engineers are a hot commodity. At the time of this writing, there were more than 100,000 job openings for data engineers on Indeed.com (roughly ten times the number of open data scientist jobs).

Perhaps the biggest downside of becoming a data engineer is that it's not one of the sexier roles in data science. Data scientists and data analysts are the ones who get to present data-driven solutions to stakeholders. As a result, they (along with the Big Data analytics experts) are the rockstars of the data science world. Meanwhile, the data engineers are working behind the scenes, making it all possible but seldom getting the same degree of recognition.

Should you become a data engineer?

That depends on what you want your career to look like. Do you like munging data more than telling stories with it? Do you find cleaning up raw data and feeding it to the data scientists surprisingly satisfying? If so, you'll probably enjoy the quiet life of the data wrangler. You'll probably also never have trouble finding a job you love at a pay rate you also love. What data engineers do is critical, and there just aren't enough of them. Becoming a data engineer is a pretty safe bet.

Questions or feedback? Email editor@noodle.com