Recruiting Real Talent Using Artificial Intelligence

Written by Rajiv Maheshwari based on his experience of creating a tool for recruiting using Artificial Intelligence (AI) and Natural Language Processing (NLP). This project was recognized for Innovation in use of Data & Technology in Asia-Pacific by Financial Times.

There is a war for talent out there. At the same time, there is an ocean of mediocrity. It is challenging for businesses to recruit the right talent for their bespoke needs. How can artificial intelligence (AI) come to the rescue in the search for real talent? Let us look at how AI and NLP (Natural Language Processing) can be leveraged to surmount the challenges in talent acquisition.

The Battleground

The traditional hiring process underwent a transformation with the advent of the internet. Job portals and socializing job openings among email groups changed the way candidates sought jobs and companies recruited talent. To a large extent, the sourcing process relies on searching by keywords. However, the status quo presents the following challenges:

Gaming the System

Potential employees have figured out how the system works and do their best to game the system. Favorite tactics include keyword stuffing without backing it up with details to get through the entry barriers. In some industry segments, falsification and exaggeration are also common.

The Bane of Change

We live in a VUCA (volatile, uncertain, complex and ambiguous) world. Sometimes, we need people to do what has not been done before. Traditional means of scouting for talent rely on finding exact matches to the requirements. If what we need is not what we have done or what people have done before, it needs a fresh approach.

Bespoke Requirements

On similar lines, sometimes we need a customized mix of roles and it does not fit a standard or a traditional Job Description.

Soft Trumps Hard

Traditional approaches are quite rigid and this inhibits the inclusion of softer factors in the screening process. It’s a vicious cycle. Since candidates know that these are not screened in a systemic manner through the CV, they do not pay enough attention to it.

Multiple Haystacks

It’s an ocean of CVs across multiple platforms that are being searched by different teams. How do we ensure consistency across people screening profiles and compare notes across recruiters? Are the methods of finding the needle in the haystack consistently being applied across all haystacks and if not, how can the process be reliable?

The Drudgery

This is often a by-product of keyword stuffing. This results in loads of irrelevant profiles showing up that the recruiter has to laboriously go through. There is simply no other way of segregating or shortlisting profiles other than going through the long-listed profiles manually.

Old is Gold

On account of the above, we tend to focus on recency, since it is very easy to filter on the age of the profile. However, the dated profiles might actually be more relevant but the search methodology makes it cumbersome and inconvenient to expand the search base.

The Big Picture

Coalescing the sum of the parts, both hard and soft factors, is almost an impossibility with the traditional approach. At best, one can resort to advanced searches that can join multiple conditions using Boolean variables (and/or/not, etc.). However, the big picture remains elusive and is uncovered only after the candidate joins!

In Search of the Unknown

Sometimes, we know that we need the X-factor. However, it is not something that can be codified. For instance, one may want an extremely creative person, but does not know how to narrow down the search using keywords in a traditional environment.

The X Factor – Getting Real Intelligence

So, let us change track from the X factor that we seek in our candidates to the X-factor that we seek in our repertoire of tools. The good news is that with the progress in the fields of AI and NLP, we can dramatically change the talent acquisition landscape.

Let us not go too deep into how Deep Learning, Natural Language Processing and their cousins have changed the world around us. However, it should suffice to highlight few key enabling powers of these tools and technologies:

Processing raw text

Historically, computational sciences focused on processing numbers and this was followed by turning attention to images through OCR. We have come a long way since then and modern tools have the ability to process raw text. This includes not just scanning and digitizing but also converting them to formats that are amenable to processing.

Encoding meaning

The journey only starts with the ability to process raw text and this serves as a building block. The next level of advances in neural networks have made it possible to encode the meaning of text, as used in our day to day language. The implications of this are massive.

In the context of our discussion on understanding profiles, we can go beyond searching for keywords and their morphological variations. We can now understand the meaning of the words and also the inter-relationship among words.

For once, the x,y and z that we used in Algebra is derived from the a,b,c that we use in our day to day communication. In other words, computers are able to understand our natural language by representing it mathematically.

However, these are only enabling tools and technologies that we have at our disposal. Let us move on from the X-factor in our arsenal of tools to the Y-Why it matters and what are the results that we can derive from usage of such tools.

Why Y – Because Results Matter

So, let us focus on the Y, a common symbol for the result or the output variable, in the context of Talent Sourcing and Acquisition. Based on my experience, the following benefits can accrue to Recruitment by using Artificial Intelligence and Natural Language Processing in the sourcing process:

If Robots Can do It, Why Should you?

The Industrial Revolution demonstrated how we should avoid manual labour in doing things that machines could do much better than us. Similarly, the Information Revolution is here to show us that we should not be wasting out previous cerebral capacities in doing mundane tasks (or not so mundane tasks) that intelligent automation can do for us.

Why Crawl When You Can Hyperloop

Traditional recruitment tools have worked on the concept of indexing based on certain attributes and keywords. However, if we develop hyperloops to transport us at 1000 km per hour, wouldn’t it be silly to crawl? Similarly, the modern technologies allow us to examine the candidature and profiles by assimilating the entire information and does not restrict the user to keywords.

I need someone just like him / her

This also allows holistic matching, rather than partial matching based on key attributes only. Very often, business leaders or hiring managers indicate that they need a candidate just like Tom or Mary. Traditional tools required a human to deconstruct Tom and Mary’s attributes. However, on the information superhighway powered by AI and NLP, one can just put Tom into the black box and get suggestions for matching clones.

You want a Unicorn, You can now get close

The magic can continue even into the mythical domain. You can create a fictional character, much like a unicorn and still get candidates that match most of what are the inherent or underlying attributes that you would have presumed while creating the fictional character.

But Where is the Gut?

I am sure your gut is revolting against the thought of allowing machines the hegemony in a process that requires intuition and your gut feeling. So, let me show you the complete picture. The gut feeling is nothing but a pattern that we learn from our own experiences. This is where extremely powerful machines that feed on more information than we can manually process, really come into their own. The machine’s ability to discern patterns is just the golden gut that the doctor ordered.

I Can See the Picture Now

Well, if you can see the picture now, it is also because of the ability to visualize profiles. We can now condense candidates and their profiles from multiple abstract dimensions to a few dimensions and attributes that can be perceived and visualized by humans. Being able to understand the picture and being able to see it, are two completely different things. Wouldn’t you agree?

Even if it’s Artificial Intelligence, It’s Real Time

Finally, saving the best for the last. We have seen how we can extend  and enhance our capabilities. The icing on the cake is that we can do all this in real time. This can be achieved, even if we decide to scan our entire resource pool, from A to Z.

Proof of Zee Pudding

And last but not the least, the proof of the pudding lies in its eating. There is a lot that modern technological advances have enabled. However, like every journey, this is work in progress. There are few inherent limitations and there are some challenges that shall be addressed in the short run. If you plan on recruiting using Artificial Intelligence and NLP, please reach out to people who have been there and done that.

In closing, I must confess that I continue to be amazed with the misuse of technology and the misguided debates. New age tools can help us re-imagine our future and our operations. However, this requires us to challenge our core beliefs and assumptions. Machines can answer the ‘How-To’ but the ‘What-If’ has to be explored by us. After all, Artificial intelligence cannot replace Natural Stupidity!