Blog – Full Width

Will Big Data Analytics and AI take my job away?

It’s that time of year again, when many of us begin to make resolutions for the year ahead. While I was preparing my laundry list of New Year’s resolutions, a report released by McKinsey Global Institute (MGI) in Dec 2017 caught my attention: “Jobs lost, Jobs gained: Workforce transitions in a time of automation”.

So, what exactly will be the impact of Big data analytics and AI? Is the automation going to eliminate most of the jobs? What we learn from the report is that by 2030, as many as 375 million workers, that is ~14 percent of the global workforce, will need to change careers or learn new skills to survive in the labor market. Looking at closer to home, in the United States alone, anywhere from 39 million to 73 million workers will be displaced due to automation by 2030 resulting in the need of up to 33% of the workforce to switch jobs.

Now you may be thinking that automation will only impact low-skilled blue-collar jobs, but they are not the only victim of this new wave of automation technologies. Stanford professor Andrew Ng, an industry expert in machine learning, says, “AI can now diagnose pneumonia from chest X-rays better than radiologists.” Many white-collar jobs including the jobs of financial advisors and analysts, insurance agents, tax preparers, sports reporters, online marketers, anesthesiologists, etc. are already in the process of automation. If you are from the software testing background you might have noticed that software testing automation has grown exponentially in the recent years. Wall Street, the media and even software developers are highly exposed to automation and are coping with artificial intelligence (AI) and automation changing the nature of their work. There is not even a single sector that is left untouched by AI. That is why Prof Ng calls AI the new electricity.

I have been thinking about an instance where such a change occurred. I still remember the rhythmic noise of clackity-clackity-clack from the typist’s desk, when I used to visit my dad’s office as a child. After the mushrooming of personal computers, word processing software made it easier for anyone to edit and retype documents. This made the jobs of typists/secretaries and book keepers extinct.

Automation is an age-old phenomenon that has existed for more than a century. Workers shifting from farms to factories, cars replacing horses, robots automating factory floors, and computers automating business processes are all well-known evidences of automation that impacted the global workforce. With the advent of self-driving cars, robots processing Amazon orders, chatbots providing online customer support, AI based fraud detection, etc. we are up for another huge wave of automation by technologies like Big Data Analytics, AI and robotics.

Do you need to panic now? Stay with me here as we learn more. To estimate the impact of the new automation technologies, Mckinsey Global Institute conducted case studies to understand the job creation and destruction pattern on two previous automation technologies: automobiles and personal computers. The research revealed that more jobs were created than were destroyed. The personal computer enabled the creation of 15.8 million net new jobs since 1980, accounting for 10 percent of employment. Jobs in typewriter manufacturing, typewriting, secretarial work & bookkeeping got displaced. But many new jobs got created including jobs in computer manufacturing, semiconductors, as well as jobs enabled by computers including programmers, IT system admin and jobs that use computers like customer service call centers. Similarly, the assembly line automation by Ford Motor Company resulted in a 10% increase in employment rate in 1915.

Takeaway Message:

People who equipped themselves with new skills had a smooth career transition whereas people who didn’t got thrown out of the job market.

Based on the lessons learned from history, though the long-term impact of automation remains positive in terms of employment, the rate at which disruption happens to workers due to AI automation will be faster than in the past. This includes the ability of machines to perform work that requires cognitive capabilities, the ability of machines to teach themselves to improve at tasks without much human intervention. Since AI automation has a potential to rapidly reallocate jobs, the authors of the McKinsey report believe that there might be a massive transition on a scale not seen since the early 1900s, when workers shifted from farms to factories. The transition would be painful for people who are hesitant to learn new skills and adapt themselves for new roles. The biggest challenge will be retraining millions of workers mid-career, says Susan Lund, co-author of the McKinsey report.

“The model where people go to school for the first 20 years of life and work for the next 40 or 50 years is broken.”

Susan Lund, McKinsey Global Institute

When I reflected upon this model I realized that it worked for my dad, but I myself had to retrain during mid-career and for my kids it is an absolute necessity to learn and retrain throughout the course of their career.

Remember the Luddites, a group of highly skilled English textile workers in the 19th century who destroyed weaving machinery as a form of protest? Their movement finally failed. We cannot simply shoo away rapidly advancing automation technologies like big data analytics, AI and robotics. There are only few job areas that are less susceptible to automation: jobs that involve genuine creativity: artist, scientist, etc., jobs that involve building complex relationship with people: nurses, a business role that requires interacting with customers, and other stakeholders, etc. For many of us who currently have a career in other job areas, the best way to survive the automation wave is to roll up our sleeves and start investing some time in learning new skills.

As I completed reading the report, I felt fortunate to be in a position where I could provide the transitional training. I finalized my New Year Resolution to create awareness and train as many people as I can in the field of big data and machine learning to help them survive in the rapidly advancing technological world. If you have been thinking about your next move or a career in Big Data or AI, ping me @SivagamiRamiah via LinkedIn or reach me through ByteQuest.Net. I will be happy to help.

Happy 2018!

Demand for Data Engineers and Data Scientists Remains High

Our research shows that employers are very invested in expanding head count in areas such as analytics and data science, product development, and sales as they strive to stay competitive in B2B and B2C markets.”  – Matt Ferguson, CEO of CareerBuilder

With data collection and data storage becoming accessible to businesses at all scale and access to machine learning tools through providers such as Amazon, IBM’s Watson and Google GCP, business of any size can harness the power of Big Data. To enable that, businesses of all size need data engineers and data scientists.

Based on Forbes here are the top 5 industries that hire Big Data related expertise: Professional, Scientific and Technical Services, IT, Manufacturing, Finance, Insurance and Retail. The below chart shows the distribution of advertised positions in the above mentioned industries.

Bigger Paychecks

It’s not just the huge demand for Big Data jobs but also the lucrative salary these jobs offer. According to Indeed, a most popular job search engine, the average salary for a Big Data professional is about 114,000 USD per annum. This is about 98% higher than average salaries for all job postings nationwide.

The average annual salary for Professionals with Big Data expertise in SFO, CA looks attractive.

Top 3 Big Data employment Markets in U.S.

As stated by Forbes, here are the top three U.S. big data employment markets:

• San Jose – Sunnyvale – Santa Clara, CA,
• San Francisco – Oakland – Fremont, CA and
• New York-Northern New Jersey-Long Island

Future Predictions of Big Data Industry

Based on IDC forecasts, a premier market research firm, worldwide revenues for big data and business analytics will grow from $130.1 billion in 2016 to more than $203 billion in 2020, at a compound annual growth rate (CAGR) of 11.7%.

Big Gap in Big Data Skills

Inspite of the huge demand for Big Data skills, there is a significant gap in terms of the availability of skills resulting in large number of unfilled jobs across the globe.

With companies hunting for professionals with Big Data expertise, now is the right time to add some Big Data skills to your toolbox and land one of the highest paying IT jobs. As you start thinking about equipping yourself to become a Big Data Engineer, there are plenty of questions that might pop-up in your mind:

• How steep will the learning curve be?
• What are the pre-requisites to learn Hadoop and other big data technologies?
• With continuously evolving expanding stack of technologies, what is the minimum required set of tools to learn to get started?
• What’s the best way to get trained in these technologies?

It is true that there is a plethora of books on big data, plenty of MOOC and online courses, blogposts on big data and innumerable discussion forums. This information overload overwhelms the newcomer and hard to know how and where to get started.

While many online based big data courses offer the flexibility for one to learn at one’s own time and pace, they provide brief overview of the subject matter, with no hands-on, real world project. In addition, these courses are designed with university-style quizzes and assignments and they don’t add much practical value.

At ByteQuest, we believe that working on real world problems is the best way to get trained in Big Data. Our Hadoop and Big Data training is geared towards gaining practical knowledge and depth required in the job market. Without toiling long hours of hard work, our students can grasp the big data technology effortlessly. Our courses are face-to-face and will be led by friendly and experienced instructors who have the expertise and passion to teach.