BRANDON.T.BANDE // ROGUE0015

HOME / THOUGHTS

Scraping 16,000 Jobs: Mapping Zimbabwe's Job Market, Part 1

· 6 MIN READ · ·

  • developers
  • zimbabwe

Hi,

I'm Brandon.

In Back Again, I promised to go a bit more tech than social commentary. This is me keeping that promise.

I was talking with a friend the other day, and we were discussing the job market for Business Analysts in Zimbabwe. The conversation went something like, "Hey, it seems like the job market for BAs is actually good." "Oh, really? How good is it?" At this point, picture me with a curious, nefarious look and one brow raised, and my colleague with a calculating gaze trying to figure it out. Long story short, you can't know what you can't measure, as one of my uni lecturers once said.

The conversation evolved into how many vacancies we could get from LinkedIn and other popular sources (at this point, we didn't have any other sources), and then into alternative sources of this data. I had previously had an idea of offering career guidance services using historical data. Being vocal and loud (typical me, of course), I brought that up, and my colleague asked the next questions: how will you find the other data? and how will you offer career guidance services?

Beyond the convo

Coincidentally, I had some free time, an unanswered question and a colleague with similar interests (a pseudo-accountability partner) at that moment. In those moments, my mind sparked like the Fourth of July (or New Year's Day in my country). I had a design problem, which would later turn into an engineering problem and then, well, everyone's problem, because I spent the next few months trying to talk about this. Key to the start of the project, I wanted the following:

  1. As much historical data as possible.
  2. To identify rising (emerging) and dying professions.

This led me to look for local job boards I could find on the internet and try to get the data from there.

First step

So we isolated a few job boards we could find and looked into the LinkedIn API, but most of what we could find did not specifically answer our questions. Then there was one source we thought we could get the information from via web scraping (basically writing a script to get the code behind a webpage and then isolate the info we want). It had to be done for every single page, which meant that for every job we had to have the link and then scrape it for the info we wanted. To be honest, we just looked and thought, oh, this could be important. However, we wanted to do it for as many jobs as possible (not just BA roles), which meant that there were a lot of pages to visit and record. To be honest, that was going to be tiring, and we were bound to give up at some point. So I decided to automate the process. During the automation, some things came up that I kind of had to think around.

1. What is the structure of our starting point?

For the final project, I utilised three sources (my colleague is more of a Data Analyst, so I opted to collect the data for her), and the sources had different structures. One was a regular HTML webpage, another was WordPress and the third was built with JavaScript. Each source needed its own approach, and that was the interesting part. One allowed BeautifulSoup and another did not, so I had to resort to Selenium. The third took too long to ingest, so I had to use the WordPress API (I discovered that after a few days of retries). There are many ways to kill a cat, and this time I had to try them all to find what worked at each and every turn. (I can already foresee so many questions around this. Can I explain more in the next article? Yes? Thanks.)

2. What do I press next, and how do I press it?

In whatever I automate, the key question I ask is, If I were to do this, what would be my natural next step? In this context, if I am on a page, either my starting point or a page I am reading, what do I do to go to the next page? In some instances, something as simple as ?page=2 or ?page=3 can help you navigate, up until the last page. In other instances, it doesn't work, and I had to find the button or element to click so that I am led to the next page (sorry, I will explain later if you are not that techy). That answers the latter question. As for the earlier one, I would simply extract the list of links (URLs) that led to the actual jobs. That led me to this approach:

  1. I find my starting point (let's say www.site.com/start-here).
  2. I list and save all the links to jobs that are there (www.site.com/job-1, www.site.com/job-2, etc.).
  3. Then I visit the specific page, www.site.com/job-1, and get the HTML.

3. The next problem

After all this, there were questions that I had to deal with, and I had to make sure that I countered them so that this came to life, and I am glad it eventually did. As a preview, the questions were: how do I store the data, how do I process it, and how do I actually make sure that I have something that my colleague could use? I am sure this is getting longer than I anticipated, so let me break here.

Stay tuned for part 2.

Let's connect via the following platforms: LinkedIn, on X as @Rogue0015, or check the Sessionize speaker profile if you want this signal live on your stage. And if you're new here, start with the talks — this blog is their echo.