How to become an expert in machine learning?

Artificial Intelligence (AI) is one of the fastest-growing industries in the world and is referred to as the "new electricity. Therefore, it is not surprising that Machine Learning (ML) and AI skills are in high demand. Over the past four years, the average annual growth rate in hiring of AI specialists has reached 74%. According to Statista there are only 28,000 AI specialists in the leading countries like the United States, 18,000 in China and 17,000 in India. 

The shortage of specialists in AI has become a factor of international competition not only at the level of corporations, but also at the level of nations. For example, China, one of the leaders in this field, spends on AI 350 times more than Russia. In Kazakhstan, the development of artificial intelligence is still at the level of programs and declarations. The competition for talent has spurred the growth of wages of AI specialists upward, for example, in the U.S. specialists with experience receive $300-500 thousand a year.

Today, many people dream of becoming ML engineers, and the demand for such specialists is very high. Verigram has faced global competition for talent in its own experience. In recent years, our specialists have been constantly getting offers from different countries, and some of their former colleagues are now working for Amazon, Facebook, and Google. 

Machine learning is a part of artificial intelligence, which includes methods and algorithms for the automatic creation of a variety of models based on data.

Verigram makes extensive use of machine learning technologies to create innovative products, so we pay a lot of attention to training specialists in this area. We have a strong engineering team, where PhD experts and doctoral students work side by side and share their experience. We attract international experts from Russia and other countries.

We asked Ainur, who has been working as a software engineer at Verigram for three years, about what a machine learning specialist should know and be able to do.

Ainur, hello! Tell us how you entered machine learning field?

Hello! I studied at Suleyman Demirel University in "Computer Science and Software" specialty. The structure of our studies involved learning the basics of computer science in the first two years. In the third year we went deeper into applied fields: there were two new subjects, among which was "Introduction to machine learning". Even in school I had an interest in futurism and I was interested in the topic of artificial intelligence. When I found out that ML was doing it, I was immediately interested. That's what brought me into this field.

What is ML in simple language?

Machine learning is a way to teach a computer to make choices without human input. If we draw a parallel with the human brain - our choices and our decisions are based on prior experience and knowledge. For example, by looking at an animal we can confidently tell that it is a cat and not a dog. A normal computer performs such tasks only if we prescribe each specific decision in advance. The task of machine learning is to get away from any specific parameters for each subject and teach the artificial neural network to replicate the work of the human brain, scaling up to handle huge data.

What are frameworks, theories, datasets, and models? Describe how it all works.

These are all basic concepts of machine learning. Let me tell you how it all works.

In order to start the learning process of a system, you need data. Preferably, there is a lot of this data. Initially, almost all of it is in "raw form" - it can contain both useful information and irrelevant information. The packet of data prepared, processed, put into one format and form is called a dataset.

From this data, the system needs to understand which data will be useful and which will not. Hypotheses and theories help us in this. Based on assumptions, we determine which data we need and which data we don't. After selecting the necessary cleaned data (training set) we train the neural network. After each training we test the model on the validation set and repeat this process in various modifications until we get a useful model which solves problems with high accuracy.

At the end of training we get a model - the final result of the program, which will solve a certain problem (for example, the identification of a dog or cat from an image) with a certain accuracy (usually more than 75%, depending on the type of problem).

Frameworks are tools that help engineers work with data, hypotheses, and models. They contain a variety of libraries and a workspace that speeds up the process by allowing algorithms and data to be operated.

Who can become an ML engineer? 

It seems to me that the closest to the field of machine learning are people with a background in mathematics (financiers, analysts) and programming who can understand the very logic of the process.

Ainur, what is the hardest thing about ML for you personally?

For me, as probably for all ML engineers, the hardest part is getting the model to a state where it produces equally good results on test builds and in real time. When we build hypotheses, we include all the theories or possibilities we know, but every day there are new challenges before machine learning and they have to be worked through. Sometimes the solution is hard to interpret - you have to spend time understanding why the model solved the problem this way and not that way.

What skills do ML engineers need and what do they need to know?

A machine learning engineer needs to know and be able to program in Python or C++, as well as a basic knowledge of mathematics, probability theory and statistics. Even if a person comes from analytics or finance, where they have worked with numbers, it will be much easier for them to understand what an ML engineer is all about.

I work with images, and they certainly relate to math. I also want to point out that an understanding of working with graphics and other media would be a plus.

Why doesn't everyone succeed in becoming an ML specialist?

I think the reason is that in ML the results of your work are often unpredictable. For example, if a developer of information system starts his work with a clear idea of what he wants to see in the end, the ML engineers do not know if the model will meet all the requirements, if it is able to solve all the business tasks. In addition, this area is now in trend, and to many it seems that it is easy - once you have trained the system and do nothing more. But in fact, it's a constant work on the selection of data, its analytics and analysis. Not everyone is comfortable with this. Most of my work is related to the dataset, not the neural network. In addition, it is quite a complicated field - to understand it you need to know mathematics, probability theory, programming and many other things, so I have to constantly learn new things. To run a system once and reap the benefits - there is no such thing in our field.

What do you like most about machine learning?

I really like working with images: I like to look, analyze, use computer vision libraries. I'm interested in figuring out how the system can best recognize certain characters. For me, working with data is very interesting.

As I said before, since my childhood I was very interested in sci-fi topics, and for me the realization that I was able to teach the computer, train a neural network and get the desired result is very inspiring and brings me back to the times when at school I read and wondered about science fiction.

What are the possibilities in ML?

You can do almost anything now with machine learning, from recognizing SEALs from images, to optimizing processes in industry and medicine.

In what fields is machine learning used?

Most often this technology is used in finance and marketing. For example, Amazon uses it to give customers a product they might be interested in. This product is selected from a million similar ones based on the analysis of data about past purchases and the preferences of other users.

You may have experienced that after searching for information about a product or service, you see relevant ads on search engines. It's Google and Yandex that apply machine learning and show you the results. ML also helps voice assistants recognize and synthesize speech. You may have heard of Siri in iPhones.

In Kazakhstan, ML is used when registering customers using facial recognition in the applications of banks and cellular operators, during calls to call centers of banks and Egov, where the voice assistant conducts a dialogue with you. You can encounter machine learning technology when applying for a loan, when the bank's customer evaluation system decides whether to grant a loan. Machine learning in Kazakhstan is used to diagnose diseases by recognizing medical images, fixing traffic violations by identifying license plates, and advising customers on websites using special chatbots.

How did Verigram help you become an ML engineer?

It was Verigram that made me a machine learning engineer. When I graduated, I planned to work in pure data analytics. But when I found out about the existence of Verigram, a company working in the field of artificial intelligence, I immediately applied for an internship. Thanks to the techlead and the team I work with, I became the person I am today. We had a lot of interesting projects in which I was able to apply my skills. Also on the projects we have taken apart many new approaches and methods from scientific articles (unfortunately, many articles did not fit or did not work in real life) in order to find appropriate solutions. Interesting problems appear every day and I really enjoy it.

What projects have you been involved in and what have you done?

I have mainly been involved in the recognition of data from authentication documents - text characters and fields (Optical character recognition, OCR). In addition, I worked with the recognition of promo codes and various symbols on product packaging. All this helped Verigram's clients to automate the routine work of the customer's staff: the employees and customers no longer had to manually retype text data from various documents and product packaging.

I also dealt with the semantic analysis (Natural language processing, NLP) where I had to compare the original description of the product with the technical specification. To do this we created a special model, which detected non-compliant data in the specification. Comparing these lists manually would have taken many weeks and consumed a significant amount of resources. Our model handled this automatically in seconds.

Amazingly, by automating such simple operations, companies save tens of thousands of man-hours every month.

What advice would you give to programmers who want to become ML engineers?

I would advise taking machine learning courses online, there are a lot of them and they are free. I advise students to do internships at companies like Verigram that specialize in artificial intelligence. It's important to work at such companies to reinforce your theoretical skills, even if it's very difficult at first. There are very few specialized AI companies in Kazakhstan, but they do exist. The experience gained during the internship will come in handy in the future and help you build your career as a machine learning specialist. Only by facing real cases in the production environment can you understand how machine learning actually works and learn how to work with it.