BRANDON.T.BANDE // ROGUE0015

HOME / THOUGHTS

Scraping 16,000 Jobs: Mapping Zimbabwe's Job Market, Part 2

· 7 MIN READ · ·

  • developers
  • zimbabwe

Hi,

I'm Brandon.

In Part 1, I left off with three questions: how do I store the data, how do I process it, and how do I make sure my colleague has something she can use? Let's pick up right there.

When things connect for the first time, it feels like pure bliss. Until you start asking yourself questions. By this stage, I had scraped close to a hundred job post pages. I had this massive dump of the pages and the code within them, and I wasn't really sure how to collect the data I needed. If I searched for keywords within the text, I could pick out the words, right? Well, yes and no. There were a few factors that made this unfeasible.

  1. Inconsistencies in data structure and format. Collecting data that's two or three years old meant there was a possibility that the structure had changed at some point. An added or removed column could make my code break or miss some info at one point or another.
  2. The same job, say teacher, could be written in different ways. Teacher, Trainer, Mentor — in some cases they were used interchangeably.

For that reason, keyword searches became a non-option, since at the expected scale the scripts would break or miss some important data along the way.

Back to the human approach

I realised that a regular human would look at the data and probably tabulate the key information. I mean, if I were to look at a job list for analysis, I would probably have columns for things like the title, the location, etc. to compare. However, it would be hard to do that over and over again, 16,000 times (18,000 at the time of writing). To do that, the key things I would need are a brain that learns what to deduce and a structure to follow as I extract the info. My skills are not exceptional enough to code such a thing, but if I were to use existing and readily available technology, LLMs would do the trick.

LLM — my system brain and how I fed it

After sitting down and thinking it through, I decided on this approach:

  1. I would extract the page code/content.
  2. Then my system brain would identify what I needed.
  3. And finally, I would put it in a table.

Easy peasy, lemon squeezy, right? Well, uhhhmmm — not really. The first part was easy to work around. I needed all the content on the job post as text. Using BeautifulSoup, that was easily done. Just one line of code and pooow, I had the entire page as text.

What is an LLM?

In simple terms, think of it as the system brain that powers AI. It is a system (model is the actual term — LLM means Large Language Model) that, after being exposed to massive amounts of data, recognises patterns and is capable of performing tasks according to the given instructions. Because of advances in AI research, a few open-weight LLMs are available for download and reuse. In my case, I would just give it a block of text, and it would identify the data I wanted and pull it out. It all worked until................. (in Ray Williams' voice, lol).

.....until what?

Until I realised that as much as it could get the information, it was not giving a consistent column name on every record. If I asked the LLM to give me a title, six out of ten times it would correctly label the column (or fieldName) as title. Then there would be jobTitle, _job_title_, _title_for_the_job_, etc. This was a problem because I wanted to standardise this data and feed it elsewhere. This prompted me to look for ways to make sure it was all standardised.

Not all LLMs are created equal

LLMs are built differently, for different purposes and for different environments. One constraint I had was my computing power. At 16GB of RAM, I had more than most, but I still had to make sure whatever I chose to solve my problem would work with what I had. I ended up choosing LLMs (system brains) with a special skill called tool calling. This meant that you could define a few tools (like small helpers that execute given tasks). You could create a lot of them, and when faced with a question, the LLM would choose the right one and make it produce the right stuff.

What was the tool I ended up using like?

All I wanted was to feed the brain a bunch of text about a job, have it extract the required fields and then present the data in a uniform format. The first part was solved. The second part was partially solved — the LLM (Qwen2.5) could pick the right tool — and to cater for the last part, I had to do two things.

I created a model (different from the model in LLM)

Phew... Brandon and models, lol. I had to create what I call a Pydantic model — an abstraction of the exact structure I wanted, with guides and constraints, so that the structure was the absolute reference for what needed to be produced. Then I created a function that would be called as a tool, to make the LLM adhere to the instructions on what was needed and then give the output as requested.

And it worked.

For the technical part, this was all the heavy lifting (thinking, in this case). As a result, I was able to read the job boards, get job listings, read them and then list them out in a table (for my colleague). For now, the technical part was done, and we were definitely heading in the right direction.

A bit philosophical

Sometimes being on the right path does not equate to arriving at your destination. Pacing, tempo and strategy come into play. Beyond the technical part, which I have articulated, there is the need to strategise the process so that a handful of extracted posts could turn into a massive structured data silo that could produce insights. This was greatly emphasised during the learning process. I got lost a couple of times, and a few things broke — not because of the code, but because of the approach. This is what the finale will touch on. (No more technical stuff, I promise.)

Let's connect via the following platforms: LinkedIn, on X as @Rogue0015, or check the Sessionize speaker profile if you want this signal live on your stage. And if you're new here, start with the talks — this blog is their echo.