Custom Trained Named entity recognizer using Spacy library for resume Data extraction

Hello all, sharing one of my experience when i was working on a project to automate the process of extracting meaningful information from Resume and cv. As largest organization receives plenty of resume per day and its become an headache for HR people to went through all the resume ….

so we build an platform which auto collect the useful information from resume and sending it to HR dashboard…

Due to company policy here i am just sharing the demo and path we followed ..and getting the available data on internet and showing result..

Named-entity recognition (NER) (also known as entity identification, entity chunking and entity extraction) is a sub-task of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc.

Dataset Collection and annotation…biggest shit

while working on this real time project it tooks pretty much time to collect the data and annotate them…

we gather almost 3000 resume and annotate them using annotation tool such as The data consists of resume which is extracted from a PDF file, followed by a dictionary consisting of a label (Name, Designation, Location, Email-Address etc.) and the start and end index of the value in the resume.

(0, 12, ‘Name’)

here for demo purpose we are taking only 200 resume with annotation available online publically

sample data contains resume and annotation part with dictionary contains entities and label

We will first load a black SpaCy english model. Then we will write a function which will take the training data as the input. In the function, first we will add a ner i.e. Named Entity Recognition in the last position in the pipeline. Then we will add our custom labels in the pipeline.

Now we are going to prepare our data for training. We are disable all the pipeline components except ner. We are only going to train ner. We are going to train for 30 iterations.

At each iteration, the training data is shuffled to ensure the model doesn’t make any generalizations based on the order of examples. We are again going to read the training data.

Another technique to improve the learning results is to set a dropout rate, a rate at which to randomly “drop” individual features and representations. This makes it harder for the model to memorize the training data. We have added a dropout of 0.2 which means that each feature or internal representation has a 1/5 likelihood of being dropped.

The model will take a lot of time to train. So we are saving the model for further use.

Now we will load the saved model into loaded_nlp_model

Now we will pass this resume to our model and see the results. We have used some formatting while printing.

Result:

Now we will test our model on an unseen resume. As the resume is in the PDF format we will extract the text from the PDF file using PyMuPDF. Then we will pass the text to our model and see the results.

Result:

You can check github repository for this:

Blog by:

Akshay Bhor: Deep Learning Engineer

an Data Scientist

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Extracting Relations Among Entities Using NLP

Support Vector Machine — with Math — part 1

How to debug neural networks. Manual.

RL — Deep Reinforcement Learning (Learn effectively like a human)

Fine-tune GPT-2

Kaggle Handbook: Tips & Tricks To Survive a Kaggle Shake-up

DOG BREED CLASSIFICATION USING TRANSFER LEARNING :BEGINNERS GUIDE

Understanding Q,K,V In Transformer( Self Attention)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Akshay Bhor ..an Data Scientist

Akshay Bhor ..an Data Scientist

More from Medium

An Introduction to Recommender Systems Using LightFM in Azure ML

English to Spanish translation with Transformer

Structural Variants on Google Cloud Life Sciences with Samplot-ML

Step By Step Customer Churn Predictive Model using ML