I trained my first tokenizer for #Balochi using my fresh newly-collated dataset. I tried things out using 🤗 Tokenizers as well as FastAI (which uses Spacy under the hood). I also published the tokenizer to the Hugging Face Hub though hit some struggles with the fact that Balochi isn't recognised as a language there 🙄

Going forward I want to better understand how to evaluate this artifact I just produced. What is it useful for? How can it be improved?

https://mlops.systems/posts/2023-06-03-training-custom-balochi-tokenizer.html

🔠 I wrote up some of what I've learned about tokenisation (with examples using Balochi). This is more of a high-level overview that tackles why we tokenise words, what options are available to us and what tradeoffs we assume by choosing one option over another.

https://mlops.systems/posts/2023-06-01-why-tokenisation.html #balochi #languagemodels #nlp

I wrote about my first steps moving forward in my Balochi language modelling project. Training a custom tokenizer is my initial short-term goal but to do that I first needed to put together a small dataset with which I could work. I detail some of the things I did to that end and a list of resources I'm maintaining as I continue on this journey.

https://mlops.systems/posts/2023-05-29-balochi-language-dataset.html #balochi #nlp #lowresource

Audio Etymologies of the Day

"Light" (brightness) comes from #Proto-Indo-European *leuk- [lɘʊk], something like this:
🔈http://www.ancientsounds.net/eastern-origins/light-from-PIE-leuk.wav

*leuk- also developed into words for "day" in Indo-Iranian languages, e.g. #Balochi روچ roʧ, something like this:
🔈http://www.ancientsounds.net/eastern-origins/PIE-leuk-to-Balochi-roch.wav

#Sanskrit रोचते rochate "shine":
🔈http://www.ancientsounds.net/eastern-origins/PIE-leuk-to-Sanskrit-rocate.wav

#Kurdish roʒ:
🔈http://www.ancientsounds.net/eastern-origins/PIE-leuk-to-Kurdish-roj.wav

#Tajik рӯз, Persian روز ruz:
🔈http://www.ancientsounds.net/eastern-origins/PIE-leuk-to-Tajik-ruz.wav

@linguistics #audio #etymology #phonetics #EnglishLanguage

RT @MarianoGiustino@twitter.com

Punto su #Iran:
- la #Rivoluzione è entrata in una fase di non ritorno
- si estende alle aree anche più conservatrici e storicamente sostenitrici del regime
- Colpiti i simboli più sacri della Repubblica islamica
- Nelle aree curde e #balochi è un inferno
@RadioRadicale@twitter.com #Turchia

🐦🔗: https://twitter.com/MarianoGiustino/status/1593759599861153792