Scraping software that support LinkedIn through API

Status
Not open for further replies.
kartikoli

kartikoli

Innovator
My company is looking for a scraping software that allow Linkedin scraping through API, we used Captain Data and Phantom buster but they can't handle our scraping needs where as my previous company was using some kind of API with there internal scraper and it was extremely fast compared to normal scrapes so I am looking for a scraper that can handle speed and workload without spending too much money. We are a small team so I can't justify too much cost for a scraper alone.

Any help with be appreciated

P.S. I don't have coding background so it will be difficult for me if the software is too technical.
 
I work closely with a team that had a similar need. A Team member (full stack Dev) created something custom for this. Sadly he has now left the org. but the tool he build was scraping posts, comments and reactions once a target account was provided.
 
I need for Lead Generation so there are 3~4 scenarios that will be needed most of the times
1. List of Accounts from SN (Sales navigator) that fit criteria
2. Account details of those companies basically we need SN ID to scrape further also the website from Company page
3. List of Contacts that fit criteria for those accounts using SN
4. Profile details of scraped contacts
5. Search Contacts when we provide name+company name or other combinations.

I've built recipes for scenario 2 and 5 (partially for 4 too) using octoparse but it can't match API speed. Not to forget Linkedin restrictions
Since this will be used by a company we can pay for the software or someone who can built something for us.
 
I had a chat with one of the software engineer from my old company and and he shared an API that we can use with Postman to fetch data from Linkedin. Sadly even after his explanation I couldn't make it work so would appreciate if someone can explain (he was in hurry anyways so didn't say much)

- This is the URL for get Request.
https://www.linkedin.com/voyager/api/identity/dash/profiles?q=memberIdentity&memberIdentity=${profileID}&decorationId=com.linkedin.voyager.dash.deco.identity.profile.FullProfileWithEntities-91

- Where profileID is a variable.
- This is the header needed.
{
'csrf-token': liToken,
}
- You can get the litoken here, in linkedin site cookles
----------
I have bunch of Linkedin urls that I need to open and grab the details so this api should do that for me but I don't know how :rolleyes:
 
Isn't CSRF (token) some cross site request forgery thing to prevent the exact thing you want to do?
 
Yup, I guess frrom the cookie if you can extract the token then it's similar to any registered member opening another member's public profile... that's why no authentication in the API.
 
  • Like
Reactions: valar007 and ibose
You can get past this limitation by using puppeteer. In brief the steps would be,

Log in to the linkedin through puppeteer > make your desired operation which emits the request > Intercept that request > Modify it, change the parameters like profileID > make copies with different profileID > and finally send them to linkedin > profit

You can expect a rate limiter on linkedin. If you are not careful, they will know you are not a normal user and you will be blocked.
 
  • Like
Reactions: vishalrao
I think it would be too technical for me as I don't have any coding knowledge :rolleyes: Regarding LN limitations I can bypass it with enough delays but was searching for a solution with speed. I know because my old company use it and if I need something done urgently I asked for help from friends.
There scraper is for internal use only else would have approached for a license from my existing company.
 
Proxies won't work well with Linkedin, they are very strict and will start giving captcha or just log you out. Though this can be pretty good for Google searches
 
Status
Not open for further replies.
Top Bottom