BRANDON.T.BANDE // ROGUE0015

HOME / THOUGHTS

Designing Vision-SL: A preview into what we considered

· 8 MIN READ · ·

  • ai
  • zimbabwe

Hey,

I'm Bee.

One might call it undiagnosed ADHD - the way I get stuck on a problem, break it apart and try to find a solution. Vision-SL is one project that brings that to light, and today I am gonna break apart what made it interesting to me. I hope you remember that in No Data, No Problem, I touched on the context of the problem. No Zimbabwe Sign Language data exists and, well, I wanted to create something that fills that gap. What's crazy about it all is that it started with simple questions and became complex product design over time.

Flirting with languages

Hello. Bonjour. Mhoro. Salibonani. Similar meanings but different languages, different compositions and constructs. In the four words above, there is no way that the contexts, compositions and constructs can be translated from one language or one form to another. Mid last year, I wrote an abstract for consideration under one of the POTRAZ funding opportunities to collect data on the sixteen official languages in Zimbabwe, and another one under a Mozilla program (Shame on me for forgetting the actual name - in my defense, I am writing offline and I might check later). All I wanted was to have a starting point for local language data to train, fine-tune and create LLMs that are fluent in local languages. Whether these failed to be considered because of relevance, my capacity to carry them out or the way I wrote the proposals - I might not know. But neither ever came to life.

Back to the basics (again)

Borrowing from neuroplasticity and how children learn, I thought to myself: what if the reason why AI can never fully speak to the local context is the fact that we try to translate from one language to another? A child needs to learn from contextual content to be fluent, rather than being told about concepts in a foreign language (not verified, my AI reviewers keep flagging that I need to support this with scholars, etc.). Upon realising that no one really cares about creating the basics (Disclaimer: from my circles) - the corpus, the infrastructure, etc. - I took it upon myself to try as much as I can. One man can change the world, as Big Sean said. The first was through the Shona words scraper. And the second was Vision-SL.

One mission, two approaches

For the corpus, I needed reference material to dissect the Shona language. I needed something that I could use to study the relationship between words, their classes, etc. And I found what I was looking for - from multiple sources (still to be cleaned, though). Then came Vision-SL, where the reference data was missing. How could I solve the case of missing/scarce data? How could I build on limited data? To train a computer vision model, I would need thousands of images in different contexts and settings. A near-impossible task given my skills (I am a shortcut dev, remember...). And on the off chance that I also needed to scale this to another language, would I be able to do that with ease?

Questions, questions, questions????

As I asked myself these questions, I tapped into my business analyst role for a moment. And my questions could be generalised as below:

  1. How do we create something that uses as little data as possible to train on Zimbabwean sign language?
  2. In the event that it is to be adopted for another language, how do we make it easier to set up and use?
  3. How do we create a system that preserves the teacher-student relationship without major disruption whilst having multiple benefits?

With each question, sub-questions started spinning, and our resolution was as follows:

  • The teacher is the absolute source of truth; let them teach, set assignments and evaluate responses/results from the students
  • Have a system that allows the teacher to specify whatever they are teaching, and the system would pass the knowledge to the student accordingly
  • Have a system that offers convenience to both the student and the teacher whilst minimising the teacher's effort and maximising the student's practice

.... and we did that (kind of...)

As a BA, I should have researched more; unfortunately, I didn't until the prototype was ready. Hand-tracking features were working and we could capture the images... I mean, we only needed a few thousand. With each save, the storage on the local machine was being tested (personal PC, remember) and the model could not accurately detect shapes for hand signals.

... and then came a question

What if we could use less space and store more? Another interesting thing we overlooked was how the system would work in production. So much image data was being stored and needed to be transferred, processed and trained on at times. For the Zimbabwean context, that was a non-starter - our system was simply conceptual. Hopefully, with time, others would improve on it.

Then came the Eureka moment. Walking to work and a bit late, I asked myself why we could not save just the hand shapes and eliminate the 60% of the data used for background, etc. So I searched for how we could auto-crop and save before sending to the backend. As I was looking into this, I also came across MediaPipe and WebAssembly. What fascinated me was the fact that both of these even solved problems I never thought I had.

From hand images to co-ordinates

Learning about how a lightweight MediaPipe model could read a hand and use 21 co-ordinates in the x, y, z axes was a game changer. I began thinking about how I could convert each hand image into those numbers. And then we tinkered a bit and discovered that with 3 saved sets of co-ordinates, the system could now pick up the signed letter from the student side (after adding three or more different letters or words). Only a negligible portion of the original payload was now being sent, so for UX, these were bonus points. Upon further research, I discovered that the model could be cached in the browser and continue working, even without the internet. Further bonus points lol.

Then we discovered that we could even make the browser do the work

Whilst learning about MediaPipe, we also learned about Edge-AI, where you have AI services doing the work at the point of interface between the user and the system (better definitions exist online). Within the browser, we could have MediaPipe doing the hand recognition and co-ordinate extraction, and then the model would be trained on/refer to the co-ordinates. This came with so many advantages:

  1. User privacy - we don't store user pictures, etc.
  2. Reduced payloads - better user experience, especially in Zimbabwe where data costs a lot
  3. Reduced server specs for eventual deployment

Then it even went a step further

As this eventually came to life, another concern came up. Offline access. For the students who really need the solutions. And we ended up designing and developing a desktop application that had offline capabilities (before my PC crashed - we are almost done again, though).

As I take the stage at DevFest Lusaka, I will not be talking about the technical layers behind Vision-SL. There are others who can speak to that better. But I will be talking about how an open-source solution, coupled with reframing a problem and research, ended up helping me to create inclusive technology and possibly helping create the backbone of sign language assistants and translators around the world. Maybe in a future article, I will dive into the limits of what I did and where I feel I still need some clarity and input to make Vision-SL better.

Let's connect via the following platforms: LinkedIn, on X as @Rogue0015, or check the Sessionize speaker profile if you want this signal live on your stage. And if you're new here, start with the talks — this blog is their echo.