Today, more and more companies are involved in Big Data and find the best way to hire big data experts. I can single out two reasons for this:

It’s easy to create large amounts of data. Even if the company does not have a significant number of customers or transactions, you can always increase the granularity. For example, save and analyze every mouse movement of the site users.

Big data analytics is paying off, and it becomes increasingly difficult to compete in almost any market without this approach.

Since everyone deals with it, we need specialists who will work with it. The number of tasks, opportunities and platforms for the development and application of Big Data projects is only growing. This means that specialists in working with big data are becoming more and more in demand.

Tasks for beginners

There are many tasks that require serious expertise and are very narrowly focused. There are projects on which only experienced professionals work. But for beginners, there is also plenty of work, because most of the tasks are typical (but no less time consuming). For example, you need to find the correct join of two tables, add several columns to a table, or change the type of columns in tables.

Such tasks arise quite often and do not require a lot of expertise. But it may take some time to agree on changes in production and to accept work. Some tasks require expertise only during the preparation of the plan. Then they can be passed on to a beginner by providing him with instructions and explaining the main features of the system used.

What are the areas of Big Data and what do different specialists do

If we talk specifically about Big Data, then all tasks can be divided into 3 areas:

• data analytics;
• development of applications for processing;
• building machine learning models.

Projects usually start with an internal or external customer – a request comes from a business unit with a more or less formalized need. For example, it is necessary to reduce customer churn or choose the most suitable tariff for each customer, or generally understand how we can automatically manage loyalty.

Once a request is received from the business unit, analytics begins. You need to answer the following questions:

• On the basis of what data should the task be solved?
• How do you access this data?
• Is it consistent?
• What result should we get in the end?
• Is there a technical feasibility of such an implementation?
• Will the proposed solution exactly meet the customer’s needs?

All these questions are answered by an analyst. Its main tools are mail, communication, and manual database queries. Manual requests are those that are not put into production for regular launch. Working closely with business units, a data analyst is the link between decision makers and technical executors of any business idea in Big Data.

Development of applications for processing big data

A data engineer is often paired with a data analyst. This is the second direction of Big Data. Data engineers are also called big developers.

Often their task is to put some kind of business logic on production rails. Those set up transformation or integration so that everything is done regularly, without creating problems and data loss. At the stage of obtaining a task, engineers often work with analysts.  Analysts translate business logic to the developer, draw up technical specifications for him and introduce the developer into the business context of the functionality being created

In general, the set of tasks for a data engineer can be defined as follows: write an application or script that will work like a clock without human intervention for a long time.