Blog of Bernardas Ališauskas

Memory is Me

2020-01-02T00:00:00+01:00

I've been slowly reaching mature adulthood as I've turned 27 just now and I've constantly poked around major meta physics and philosophy question as most of people have. While I hadn't found many answers yet one thing I've got certain - how valuable memories are.

Objectively speaking our memory structure is often cited as one of the reason for our species intelligence and consciousness - an idea that has been explored in many books, shows and movies. I fully agree with this idea - what are we if not a biological machine with a collection of experiences and memories.

Unfortunately as I started to get older I feel I'm starting to forget things. Some might be pure biology - I'm vegetarian so I often skimp on B12 and my diet is lazy on iron - but most of it can be blamed on method. By that I mean my current methods of preserving memories are flawed and in need of improvement.

So this is my new years resolution - improve on my memory management!

Understanding and Improving Memory

To improve something you have to understand it - well at least a little bit. For that I've went through few resources and got a worthy summary.

Unfortunately you can't beam your memories as radio signals just yet :(

There are many kinds of memory, but the one I'm interested for this topic is long term declarative memory. Without digging to much into this immense subject declarative memory refers to all memories that are consciously available - things you can remember on demand. This is further split into:

Semantic: knowledge about external world - facts.
Episodic: memory to specific event in time.

Episodic memory is what really interests me and especially how to improve it but semantic memory has been a bit of an obsession of mine for a long time too, so I'd like to investigate both of these mediums.

Improving Episodic Memory

According to bunch of resources there are ways to improve episodic memories but nothing is really certain so I've designed myself few episodic memory improvement strategies that I'd like to follow through this year.

It seems that sometimes brain needs a bit of hand to start going!

Emotion

Generally memories with emotion attach to them tend to be much stronger. Obviously you can't really fake emotions to improve retention of a memory, but there are few ways you could take advantage of this fact.

One particular idea of flashbulb memory indicates that attaching feeling of scale and importance helps to retain a more vivid memory:

A flashbulb memory is a memory laid down in great detail during a highly personally significant event, often a shocking event of national or international importance.

Taking advantage of this idea in this context is a bit hard but I've noticed that merely acknowledging emotions attached to memories helps to strengthen episodic memories as well as make them just more pleasant. For example a memory at the beach - I was happy and dazzled by the scenic view but the abundance of jelly fish made me feel a bit anxious. Just acknowledging these emotions makes the memory more vivid, it's as if the emotions helps the memory to branch out to deeper parts of me.

For this section the resolution is quite simple - acknowledge the emotions attached to memories.

Drugs

The relationship between memory and drugs has always been quite conflicting but there are some clear evidence that drugs can improve it.

A 2015 meta-analysis of high quality evidence found that therapeutic doses of amphetamine and methylphenidate improve performance on working memory, episodic memory, and inhibitory control tests in normal healthy adults.[1]
[1] - https://repository.upenn.edu/neuroethics_pubs/130

Unfortunately most of episodic memory enhancing drugs are quite hardcore and unsurprisingly are developed to fight against various serious diseases like Alzheimer's or Parkinson's. I've tried digging around but every episodic memory drug seems to be quite hardcore messing of proteins and neurons which means any sort of recreational or supplemental usage is not worth it.

However malnutrition is clearly attributed to episodic memory (and memory in general) quality. Clearly not many people reading this blog are not suffering from malnutrition, right? Well I've been doing yearly blood tests and taking along everyone I could talk into it with me and you'd be surprising - almost everyone was missing something. Personally my last years test was low on iron, B12 and vitamin D - I could clearly see my memory improve after sorting this out within few months.

My resolution for this section is simply to take care of nutrition better - more varied diet, some supplements and general awareness that the biochemistry in my body is important for my self.

Recall

By far the most common way of strengthening episodic memories is recalling the memories and everything attached to them. Basically the more you remember the memory the more it gets stuck in your head.

I'll be focusing on this the most in my approach - designing exercises to efficiently and accurately recall memories seems like the most effective improvement strategy.

Being a software nerd that I am I'll start my subject with a bunch of software tools that can help.

Monica - Personal Contact Assistant

First I'd like to start out with Monica - a personal assistant web app where you can create contacts and keep track of experiences and events.

it's like a digital rolodex!

This tool is not only great for recalling memories but for managing contacts as well. It helps you to be a better friend, coworker or even a family member - historical information makes it easier to care!
I've already been using Monica a bit but my goal here is to incorporate it even more into my routines.

Photography and Video

Second tool so to speak is simply taking more photos. I love photography and there's an immense memory capture value here. Just scrolling through instagram stories or snapchat archives makes for a really fun and fulfilling experiences. Needless to say capturing visual moments is a brilliant recalling tool and I want to do more of it.

Take pictures of everything and sort it later!

My currently tool usage for this medium is as follows:

Instagram Feed - feed of my own photos, it's more of a collage or a CV of my past self to my future self.
Instagram Stories and Snapchat Archives - feed of my activities, snippets of various events and adventures.
Photography on Pixelfed - most interesting photography experiences go here.
Photos and Videos - Feed of my bigger events. I like to edit videos and photos but it's rather time consuming so I'm reserving bigger events.

For storing photos and videos on cloud the biggest challenge is actually curating the media to make recalling a streamlined and pleasant experience.
For that I'm using Pcloud cloud storage and dedicate few minutes every weekend to clean up auto-uploads for a more pleasant recalling experience if I do want to go back and look through my photos.
There are applications which help to curate your feeds by generating stories of what it thinks are most important bits - this helps immensely if you don't curate your albums by hand. I haven't researched this thoroughly enough so one of my resolution goals is to categorize my photos better. For that I'll be investing more time with Darktable 3.0 release and go from there.

Journal

An obvious way to strengthen recalling is to start a journal (aka diary or log book). Not only it can be therapeutic way to relieve your mind and organize your thoughts but it's a great tool for improving memory too. Recalling memories right after they happened is a sure way to strengthen them, to add you can easily revisit these memories and recall them at any time given that your logbook system is properly indexed.

Is making a journal terminal with raspberry pi going too far?

I love static site generators and pelican in particular thus I've started my log book as a local static website. Pelican allows me to categorise my entries by tags, date and any other additional meta fields allowing easy recall to any specific entries.

html journal with pelican

For this I've developed pelican-journal - a pre-made pelican project designed for tag-based journaling.

Exercise, Meditation and Fasting

There a sufficient amount of evidence that exercise, meditation and even occasional fasting helps to improve memory. However I've struggling integrating exercise into my routine.

Often I find exercise exciting at first but lately it has been failing to enter my routine. I identified few causes:

Lack of visible progression. This is a bit difficult to address as the progression tends to slow down after initial two months or so during which you can upgrade your weights every few visits. After a while, however, I feel that I'm hitting a barrier of X kilos and lose interest.
For that my resolution is tracking the progression in notes: weight, diameters and strength.
Pushing for too long gym visits.
I've been a bit of a victim of peer pressure to stay in gym longer than I really want to which really just kills my interest. I find 45 minutes to be a sweet spot for me.
My resolution in this case is to push myself less.
Timing.
This has been the hardest to figure out. When to go to gym? Sometimes I enjoy mornings, sometimes I enjoy evening but constant mixing of gym times messes up my routine.
For this I've decided to focus on 4-5 o'clock. Hit the gym for an hour and just in time to catch early Thai sunset for after-gym recreation.

For mediatation but I've been struggling to find enjoyment in it. I usually pick it up for few weeks and then lose interest. I've tried apps like headspace without much luck. However it only takes 5-10 minutes, I really need to give it a proper go.

Summary and Further Research

For 2020 I've decided to strengthen my episodic memory through: taking more pictures and organizing them better, keeping track of contacts and events using Monica assistant and keep writing daily or at least semi-daily journal.

There seems to be a lot of information on the internet and I'm still on the lookout for possible improvement strategies but so far I'm satisfied with few approaches I've described here. For further improvement I have my eyes on magnetic memory method - seems like memory course but the author seems to be quite legit their article on episodic memory was surprisingly well writen, though I'm not very trustworthy of self-help salesmen and the name "magnetic" doesn't give much credibility but based on that article it might be worth investigating further.

Happy 2020

Whoever read this, enjoy the new year! Hopefully you found some useful material in this article to make this year better :)

Get Organized With TaskWarrior!

2019-10-26T00:00:00+02:00

If you've been part of terminal ecosystem for you're most likely familiar with numerous terminal todo sort of programs. I'd like to introduce you to king of all of them Taskwarrior.
The name might sound intimidating but don't go - it's quite the opposite: it's butter smooth!

Intro

Your key command here is task. Typing this alone will list your open tasks:

$ task

ID Age  Description                    Urg 
1 32s   write a blog about taskwarrior    1
2 11s   make carrot cake                  1
3 2s    pay electric bills                1

Then the rest of cli goes as intuitive as it follows, these are your 3 vital beginner commands:

add "some task" - to add a task.
done <id> - to close task.
modify <id> - change something about task.

Finally Taskwarrior has one vital important feature for managing your tasks - tags. You simply specify tags with + signs in front of your commands:

$ task +work add "fix linkedin.com crawler"
$ task +work 
ID Age  Description                    Urg 
1  2s   fix linkedin.com crawler        1

You can see already with these 4 little things: add, list, modify, tags - you have great powers. Time for example workflow!

Example Workflow

This is a simple version of my taskwarrior workflow; First add tasks the moment you get them and try to split them up into small tasks - you're not an idiot you won't get confused don't worry:

$ task add +work "fix project1"
Created task 1.

$ task add +work "msg admin about project1 dump"
Created task 2.

$ task add +shop "potatoes - for potato cannon"
Created task 3.

$ task add +me "1 chapter of new book"
Created task 4.

$ task add +me "wash dishes"
Created task 5.

$ task add +gf "research christmas present on etsy"
Created task 6.

$ task add +me "practice bass a bit"
Created task 7.

$ task
ID Age  Tag  Description                        Urg 
1 7min work fix project1                        0.8
2 6min work msg admin about project1 dump       0.8
3 6min shop potatoes - for potato cannon        0.8
4 5min me   1 chapter of new book               0.8
5 5min me   wash dishes                         0.8
6 4min gf   research christmas present on etsy  0.8
7 4min me   practice bass a bit                 0.8

Right, got bunch of tasks and now just pop them as you feel like. Having a weekday work day?

$ task +work
ID Age   Tag  Description                   Urg 
1 11min work fix project1                   0.8
2 10min work msg admin about project1 dump  0.8

Task 1 really needs to get done:

$ task start 1
Starting task 1 'fix project1'.
Started 1 task.

All this does is mark the task in your list that it's "ongoing" :

$ task +work
ID Active Age   Tag  Description                   Urg 
 1 1min   14min work fix project1                   4.8
 2        13min work msg admin about project1 dump  0.8

Been working hard for an hour or so, time for a short break. Lets keep the task active so we could jump right back in after our break and take a look at me tasks we have for our break:

$ task +me 
ID Age   Tag Description           Urg 
 4 14min me  1 chapter of new book  0.8    
 5 14min me  wash dishes            0.8
 7 12min me  practice bass a bit    0.8

Great time to boot up the bass for a song or two and while Rocksmith on ps4 is booting up - wash the dishes:

$ task 5 done
Completed task 5 'wash dishes'.
Completed 1 task.

Break over, simply jump back to work, the task you've been working on is still glowing red there waiting for you ;)

Get Micro Motivated!

Taskwarrior allows this sort of dead-simple micro-motivation. It's so simple an intuitive that in never feels like additional work.

Often it's easy to go through your day without having tangible experience of your accomplishments. No matter how small your tasks are - the day feels so much better when you can visualize and feel the tiny hills you've conquered!

1 Year in Chiang Mai, Thailand

2019-06-02T00:00:00+02:00

I've been living in Chiang Mai - biggest city in North Thailand - for over one year now. I did a lot during this year and experienced many things. The food is great, the housing is cool and the language is complicated. Let me tell you all about it!

Visa And Getting Here

Getting to Chiang Mai is dead easy - take a flight to Bangkok and then another one straight to Chiang Mai! There are flights almost every hour and are surprisingly cheap. The flight from Europe (e.g. Riga) - Bangkok is around 350€ + Bangkok - Chiang Mai 50€ = ~400€.

For Visa it's a bit more complicated. There are several types of visa but the most interesting ones:

Tourist Visa

Anyone can get this visa. You can either get 30 days on arrival visa or submit your passport with ~50€ and get a 60 days visa with a possibility of 30 days extension for another ~50€.

There's one thing to note though. For tourist visa you need:

Onwards Ticket - some transportation ticket out of the country
Financial Proof - that you have some money (bank statement or cash)
Hotel Reservation/Address - this one is rather weird as you can just enter random hotel address.

You really don't want to mess with a guy that spends 8 hours a day in a chair stamping documents!

I've started with this visa one year ago by taking my passport to Thai embassy in Hanoi and getting it back with a visa next day -> easy peasy!

Education Visa

This is visa you can get by applying to study something. This can be almost anything but usually it's either Thai language or self-defense training like Thai boxing (Muai-Thai).

This is actually very cool idea as it setup this little education infrastructure for foreigners. You can stay here as long as you study something of ours! Pretty awesome!

There are either proper schools or "visa farms" that really don't do any real teaching and only provide you documents to get your visa.

The great bit about student visas that they are kinda "eternal" (some suggest that after 3-4 years you might start to be dnied) - as long as you are studying you can extend your visa for 90 days for ~50€. It only takes few hours of your day to get it done and you're set!

Getting My Education Visa

I've picked up the best rated school in Chiang Mai called Mana Thai School, dropped ~750€ for a yearly course which involved 2 hour lessons 4-times a week and found out a rather bizarre initial process.

With school contract in hand and tourist visa in my passport I found out that to acquire student visa I need to leave Thailand and apply for it in Thai embassy - here started my unexpected travel to Laos!

flight->bus->tuk tuk->bus->taxi->flight

This is called visa run - you leave the country to apply for a new tourist/education visa. The most common one is going to Thai embassy in capital of Laos - Vientianne and it's awkward, stressful and surprisingly expensive journey.

It took me two flights, couple of bus rides and one expensive taxi ride to not miss my flight back. I also got ripped off at the border by being required to buy visa because the workers didn't know what Lithuania was.

waiting for the wrong embassy to open

Finally I went to the wrong embassy early in the morning, so when I finally got to the real one I spent 8 or so hours in a queue to get my passport and request to study submitted.

there is some interesting sight-seeing to do!

For the rest of the day I explored Vientiane which was quite tiny but cosy. It's was a pleasant mix of Thailand and Vietnam but not as exciting, busy or captivating like either. After exploring Pagodas, historic buildings, parks and markets I had enough but I'd definitely would come back to spend there a week or so.

Studying Thai

I've spent almost a whole year learning Thai and it was quite fun. Thai is a very unusual language for my background - it's my first tonal language and it's quite a barrier!

I've been studying 2 hours a day for 4 days a week in a group of 5-10 people at Manna Thai School. The first two months we learned just plain talking and some writing in latin'ish alphabet - it was great fun. After writing and reading started and things got much more complicated quickly.

every letter has an official word attached to it. Like Z for Zebra but standartized which is really great for learning!

Thai writing system is very unique and fun but also very dated and filled with old rules, exceptions and dead letters. Some interesting facts about Thai language:

Thai language is a tonal language and has 4 different tones, which means your tone can be the difference between a horse or a dog or between number two and a brothel. Tones are really hard to as rules are complicated and even most native thai speakers don't know them and just talk by ear.
There are 46 consonant letters in Thai but only 21 consonant sounds, which means you have multiple letters for same consonant and they are not interchangeable, i.e. you have to use correct letter T (there are 6 T letters) from in word Turtle or you might end up writing something else entirely.
There are dead letters that are either completely dead or used in very few words.
Thai vowels can go in any direction: left, right, up, bottom! Unlike most languages that simple go from left to right in order thai is very peculiar and instead you write consonant and attach 1 of many different vowels that have fixed location around the consonant (e.g. long U goes under a consonant while short a goes above it)
There are no spaces between words in written thai which makes reading as a beginner really difficult.

Words "I love you" writen in Thai, sounds something like "pom rak kun" (from male's perspective)

All that being said it's a fun language and I enjoy learning it, it's just not a very good one and definitely in need of some reforms. Neighbouring countries of Laos and Myanmar use similar languages with few reforms and they are known to be much more sane.

It's a great deal; 400 hours definitely seems a lot!

On paper I did 400 hours of classes but I'll be honest - I've been skipping a few classes. That being said I got quite a bit done. I'm quite satisfied with my current elementary-school level writing/reading and some basic talking/listening. I think I can pick it up to a rather fluent level in another year. according to FSI source to acquire basic fluency in Thai you need around 720 hours. So I'm trying not to beat myself up too much.

Main problem is actually practicing Thai. It's surprisingly difficult to immerse yourself in the language if you don't actively put effort into it. I've been working on extending this with some training apps and tv shows on Netflix.

Overall Thai language is fun and I'm glad I picked it up. It's unusual writing system, tonal pronunciation and word order is a great mind exercise!

I'm in the works of writing a blog about all thai language learning tools I've tried, there are few good ones!

Lodging

In this year I lived in three different condos and all of them were great value! The quality condo pricing ranges from 270 - 550€. If you want nice location, pool, gym, rec space(like library or park) and a bit more space then 500€ with a yearly contract is a fair number - and that's an absolute steal!

All three condos I lived in all had pools, great staff, security and great location!

Prompt Condo - had great location was cosy but not very luxurious - 420€/month. It has a cosy pool on ground floor and a little gym right above it. No shared space like library or a garden though.

pool and a gym on second floor. It's quite cosy

Play Condo - my favorite experience in Chiang Mai. It's not very big but the staff was always helpful, the room for 560€ was pretty huge (50m² or so) and the rooftop pool and library is a real pleasure. The location was close to perfect too - across the street of night market and a super mall.

rainy day near the pool is perfect for relaxing!

Escent Condo - I had to move to a different town area and found this new condo that is 26 stories high and is pretty much empty. For 460€ you get around 40m² of living space with nice view plus a rooftop pool/sauna, libary and gym. It's right next to huge super mall and few local markets too. The only down-side is that it's a bit far from a city center.

26th floor really delivers a nice sunset!

I found the condominium infrastructure very impressive and enjoyable. I'd have trouble going back to european style flats and for this price it's worth staying here just for the housing!

a perfect spot for an afternoon nap!

The general cost is very affordable. You can find similar condos in 300€ range too if you're more flexible with location. It's worth noting that electricity bills can add up really fast so you should add 30€ or so for air-conditioning alone!

Food

This segment probably deserves a blog post of it's own but to put it shortly thai food is great and surprisingly supportive of vegetarian diet, though egg is very prevalent so vegan diet is a bit harder.

Thai food is very rich in taste, be it salty, sweet or sour. It's relative healthy too - maybe a bit heavy on saturated fats (oil) and carbs (rice) but if you can pay a bit attention you can have a very healthy vegetarian diet without much effort!

7€ for two days worth of delicious, vegetarian food!

The biggest appeal is sheer value. I often take a backpack full of plastic containers and fill them up with various dishes at a local vegetarian restaurant. For 8€ and can eat quality, delicious and somewhat healthy food for 2-3 days! The best part is that it saves a bunch of time too!

Lastly being vegetarian in Thailand isn't hard at all but some effort is required. Thai people really like to eat meat so they tend to mix some meat into every dish which is hard to avoid unless you can communicate with staff to adjust the dish for you which they don't mind doing. The word for vegetarian is เจ (pronounced jėė) which stand for buddhist vegetarian diet.

signs used to indicated vegatrian (jee) restaurant

The easiest way to find some delicious, cheap vegetarian food is looking for this word (usually in yellow background) around Temples - the restaurants/cafeterias serve the best vegetarian thai food there!

Produce

Vegetables and fruits in Chiang Mai are highly seasonal, though mostly by price and quality not by availability. There are two seasons in Thailand - wet and dry but for plants it's a bit more complicated. There are these, as I'd describe, "surge" months where specific vegetable or fruit is really kicking off and you can just drown yourself in it. As of writing of this blog the Lichee season is starting and thai lichee are absolutely delicious!

Produce stalls are everywhere! [1]

The local markets (which are everywhere) offer a lot of quality produce for pennies - especially if you go out a bit out of town or go to wholesale markets. Some produce is a bit harder to get by at times but can be found at bigger markets or at some import shop.

One of my favorite produce related experience is that often local alleys have small stalls of good quality produce, so if you're in a sudden need you can just pop down from your condo and grab some carrots or whatever you're missing for that delicious soup your cooking!

For other favorite experiences here's few of my favorite produce from top of my head:

Mango - green mango is pleasantly sour and great snack dipped in sauce or spice mixes, while yellow mango is perfect sweetness on it's own.
Baby mango - it's just young green mango but you can eat them like apples, they are quite sour and very refreshing. Usually people eat them with sweet tamarind sauce.
Lychee/Longan - there are many kinds, some sweeter some less so but all of them are delicious and super cheap when in season!
Guava - apple of thailand. Packs more vitamin c than a lemon!
Pineapple - Chiang Rai (a town north of CM) is probably the world capital of pineapples. There are so many kinds and many different tastes! The baby ones are especially delicious as they lack of protein that burns your mouth.
Mangosteen - these little guys just melt your mouth! They are quite seasonal and difficult to peel but are absolutely worth it!
Salads - if you're into salads CM got you covered, I'm eating Red Coral almost daily! There are also many salad bars and make your own salad restaurants like Okaju and Salad Concept where you can get bowl of delicious, dressed salad for 2€!
Eggplants - there are lot of different eggplants and they're all delicious. If you know your way around a kitchen you can really go wild with these guys!
Berries - there aren't many berries in Thailand but few that do exist are extremely delicious. Mulberries and Cape Gooseberries are to die for! Unfortunately just like everywhere berries here are highly seasonal too.
Mushrooms - there are many different mushrooms in varying tastes but all of them are delicious! My favorites are just sauted or fried with butter and chilly powder. Eringi mushroom is my daily protein driver!

Another absolutely delicious use of produce is pickling. You can find almost pickled anything - mango, tamarind, grapes and all sorts of plume-like tiny little fruits I don't even know the name of!

Add some chilly powder and sugal mix and you got yourself a perfect snack!

All that beings said some produce is surprisingly disappointing. Tomato is my favourite fruit and thai tomatoes aren't great, they are never proper ripe and aren't as watery as imported ones. Another disappointing thing is serious lack of canned legumes, beans etc. - it's awfully expensive and not as great as you would get back in europe.

Weeds/leafs/flowers

There are so many little obscure plants that Thai like to deep or stir fry or mix in some curry. I could hardly do it justice in this small section but variety is truly astonishing.
If you take a thai cook to a local jungle and ask them to start pointing at edible plants you'd be surprised to see them turn into a helicopter.
Stir fry pan is probably the ultimate survival tool for thai outdoors.

The most basic but absolutely delicious dish is simple omelet, mixed in with some of these obscurities eaten with some chilly sauce and rice. It's has been an easy breakfast/lunch for when you want some heavy food - pop in to veggie stall at your alley to get some plants and sticky rice, toss the plants in an omelet and combined that with some sauce and sticky rice you have a carb and fat rich, delicious, and most importantly - lazy, lunch!

Another one of my favorite dishes called Nam Niau is a soup made with dried flower called Niau, literal dish translation water Niau and it's delicious! usually combined with rice noodles, tomatoes and some extras like liver or in vegetarian versions mushrooms.

Niau flower soup is different everywhere I tried but the core soup is always as delicious as you expect it! [1]

Coffee and Tea

Thailand is full of Tea and Coffee plantations and they grow some of the most delicious stuff in the world! Not only that but these plantations tend to be a fun place to visit and often have a little place to get some dinner with a view.

an hour drive to the mountains for breathtaking lunch over a tea plantation

Thailand really shines in this regard, some of the best Arabica coffee in the world can be found here in North Thailand. The mountains are just tall and accommodating enough to produce great quality beans that are further roasted and taken care of families and hill tribes that have been doing this for generations.

The coffee is really cheap too if avoid the tourist brands. A 500g pack high quality arabica from a hill tribe (you can even find some in nearby super market) will only cost you 3€!

The cafè scene in Chiang Mai is very healthy too. There are a lot of little coffee shops sprinkled around even at the most remote corners of the country. All serving good, delicious coffee 1-3€ and often provide an either themed or cosy to sit down to drink it.

my favorite café "Art Roastery" has geese and other birds keeping you company

Since I work remotely I often do that from a café. The best cafes in my experience are located towards Doi Suthep mountain, so if you enjoy café hoping that's the direction you should aim at.

The Smog

So far most of the things I said about Chiang Mai were very much positive. It's absolutely true - Chiang Mai is wonderful for the bigger part of the year but starting around January burning season starts and goes on until late May. It's terrible.

March compared to end of May

It's ugly, but not only that it's just awfully unhealthy and the smog has a very unpleasant smell which makes any outdoors activity really unpleasant. Burning season is also high season with more tourists coming from cold countries to enjoy Thai sunshine - means prices are often higher and the town is packed more than usual.

Combined these two things I couldn't recommend anyone coming to Chiang Mai during the burning season, in fact I'd recommend leaving it for South Thailand or neighbouring countries.
That being said some few people are absolute fine with it so personal tolerance definitely plays a role here.

Leisure Time and Conclusions

There are a lot of things to do in Chiang Mai, most of it involves the beautiful nature of North Thailand but there's good amount of entertainment too. Chiang Mai has many shopping malls that provides relatively cheap and enjoyable shopping experience, there are some activities and events happening every week. There's a healthy music scene and a sizable expat community for various activities ranging from board-game evenings to book and yoga clubs.

cosy little lake almost at the city center

To summarise Chiang Mai is a really fun, affordable and cosy town. The housing is amazing so is the food and nature. It's a great place to live in for 8/12 months and you really have to hermit your way through the 4 months of burning season (or ideally leave) but this can be planned around.

The year I spent here was really enjoyable with the exception of burning season. I really enjoy North Thailand nature and there's always some new place to explore. I got driver license, scooter and a car which I believe are vital to truly enjoy the exploration and nature aspect of the area (maybe you could do without a car but with big bike instead).

I plan to stay here longer so there are many more blogpost to come! I plan to cover some subjects like expenses, food and my favorite activities in greater detail.

For more Chiang My photos check out my pixelfed feed.

Credits and attributions

1 - https://commons.wikimedia.org/wiki/User:Takeaway

Diving Into Web-Crawling

2018-12-10T00:00:00+01:00

Web crawling is a brilliant source to bootstrap your application. Almost every application requires data of some sort and why not just pick up some public data available on the world wide web!

In this introduction I'll cover the core ideas behind web-crawling and web-crawling with python.

What's Web Crawling?

First it's important to wrap your head around the stages of web crawling program.

crawler or spider A program that connects to web pages and downloads their contents.

To put it simply it's a program that goes online and finds two things:

data the user is looking for.
more targets to crawl

e.g.

Go to http://shop.com
Find product pages
Find and download product data such as price, title, description

Stages

A usual web-crawler program will be made from 4 core stages:

Discovery - find product urls to crawl
Consumer - consume product urls and retrieve their htmls
Parser - parse html data into something useful
Processor - process the data with pipeline, filters etc.
Loop!

note: these stages don't have to have to happen in strict order

There can be few more steps in between but the core logic of a web crawler looks like this:

worth noting that websites can return all sorts of content not only html. Some return json, some just text and some code like javascript.

In very simplest python pseudo code for a crawler would look like this:

url = 'https://www.python.org/jobs/'

job_urls = find_job_urls(url)
data = consume(job_urls)
data = process(data)

import json
with open('output.json','w') as f:
    f.write(json.dumps(data))

You can find full crawler at the end of blog post.

Crawling

There are two ways to crawl http urls:

Synchronous - slow and simple
Asynchronous - complex but blazing fast.

Which one to use? As beginner or if you don't need speed stick with synchronous approach. The programming is much more simple and easier to debug. However asynchronous knowledge is very valuable and learning it is a great idea

Synchronous

This is very straight-forward approach. Everything goes in your program's order:

for every url: download page -> parse it -> store it -> repeat

For synchronous crawling most popular library is requests

However while this approach is simple and easy to maintain it ends up being very slow as every time program connects to a webpage to download data program need to wait for it to response - in the mean time program could be doing something else: like download another webpage, parse the data or store it.

when is synchronous approach good enough? Often a lot of applications don't need to retrieve a lot of data (e.g. football match score crawler) then sync code is more than enough.
Alternatively a lot of website have request limits that are high enough that match synchronous code slowness.

Asynchronous

Async programming is a bit more complex to wrap your head around but to put it shortly it allows the code to be executed in pararel.

Your program can schedule 100 request to a website at once and handle response as they come in or do something else!

So it terms of speed it vastly outperforms synchronous crawlers as they don't have to wait!

"checking" square here is very simplified representation of async mechanism

For asynchronous crawling there are a lot of choices and no clear defacto standards. I recommend requests-futures for caroutine based approach and twisted or treq for callback based approach.

caroutines or callbacks? While caroutines are much more favored async principle these days callbacks have a special place in web-crawling community as the logic tends to match scraping patterns better

Parsing

There are all sorts of data types on the web, but most likely you're either will be crawling html or json.

Html

Html is a subset of xml tree structures. It's a great data type for representing structure, however it's not a great data type for representing data itself.

    <html>
        <body>
            <div id=name> rubber chicken </div>
            <div id=price> 55.99 </div>
        </body>
    </html>

So usually this data is either converted to database tables or json, csv documents:

csv documents:
```
name,price   
rubber chicken, 55.99
```

json documents:

[
    {
        "name": "rubber chicken",
        "price": 55.99
    } 
]

database tables

------------------------------
|       name       |  price  |
------------------------------
|  rubber chicken  |  55.99  |

What data type to chose for output? Anything works! However some data types are easier to work than others. json is an easy to format to work with as it translates to python dict seamlessly. csv is great to work with as it's an easy format to write and parse. json-lines format is best of both worlds. What about data bases? Document based databases are often preferred when web-scraping. Like MongoDB and couchDB, they are great for storing json data, while relation databases are a bit less straight-forward but come with their own benefits (like sqlite and mariadb)

Parsing html

Since html is a structural data type we can parse it quite easily. For that there are xpath and css selectors and appropriate python libraries that implement this selector logic.

For python all you need is parsel which allows you to use both types of selectors to parse data. Alternatively you can also use a popular alternative beautifulsoup4

Here's a parsel example:

from parsel import Selector

html = """
    <html>
        <body>
            <div id=name> rubber chicken </div>
            <div id=price> 55.99 </div>
        </body>
    </html>
"""
selector = Selector(text=html)

# css selector
name = selector.css('#name::text').extract_first()

# xpath selector
price = selector.xpath('//div[@id="price"]/text()').extract_first()

print(price, name)
# ' 55.99 ',' rubber chicken '

Surprisingly simple!

Which one, css or xpath? Generally css selectors are much easier and more bautiful but xpath selectors are much powerful. So ideally use css and fallback to xpath when encountering something more complicated.

Parsing with regex? Generally parsing html with regex is a bad idea as regex patterns will quickly become unreliable. Html is structure data type - embrace it!

Json

This type of data is often used by website internally together with javascript. Sometimes you can crawl these pages directly either through public or internal websites API.

It's super convenient as you don't need to do any parsing yourself!

Parsing json

Parsing json is super easy as it can be read as python dictionary right out of the box:

import json
json_data = '{"data": [{"name": "product", "price": 55.99}]}'
data = json.loads(json_data)
print(data['data']['price'])
# 55.99

These days more and more web pages become dependant on javascript thus often providing json data

Synchronous Example

Lets write a simple article crawler for python blog posts!

for this we'll use parsel and requests packages. You can get them via pip:

pip install parsel requests

Our spider logic would follow:

Go to page with all links to blog posts
Go to every blog post
Extract data
Store data to file

import json
from urllib.parse import urljoin, unquote

from parsel import Selector
import requests

session = requests.session()  # 1


def discover():
    """Discover job urls in jobs listing page"""
    jobs_url = 'https://www.python.org/jobs/'

    jobs_html = session.get(jobs_url).text
    jobs_sel = Selector(text=jobs_html)

    urls = jobs_sel.css('.listing-company-name a::attr(href)').extract()
    return [urljoin(jobs_url, url) for url in urls]


def parse(url, html):
    """Parse job html"""
    sel = Selector(text=html)

    company = sel.css('.listing-location a::text').extract_first()
    email = sel.css('.reference.external::attr(href)').extract_first()
    title = sel.xpath("//h2[contains(text(),'Job Title')]"
                      "/following-sibling::text()").extract_first('')
    description = sel.xpath("//h2[contains(text(),'Job Title')]"
                            "/following-sibling::p/text()").extract_first('').strip()
    return {
        'url': url,
        'location': company,
        'title': title.strip(),
        'description': description,
        'email': unquote(email.split(':')[-1]),
    }


def process(data):
    """Here usually you'd add extra processing steps
    like upload to database
    or adding crawl time
    or determening whether to drop results based on their values"""
    return data


def consume(urls):
    """Consume job urls by downloading them, parsing data and saving to disk"""
    for url in urls:
        print(f'crawling job: {url}')
        html = session.get(url).text
        yield process(parse(url, html))


def crawl():
    """
    Complete crawl loop:
    1. Discover job listing urls
    2. parse html for data
    3. process data
    4. store data
    """
    urls = discover()
    data = consume(urls)
    with open('collected.json', 'w') as f:
        f.write(json.dumps(list(data), indent=2))


if __name__ == '__main__':
    crawl()

As you can see the crawler is split into 4 parts:

Discovery - discover job urls in listing page
Consumer - crawl job urls
Parser - parse data from job htmls
Processor - process data and save it to file

1 - connection sessions establish connection to server and keeps it open for more requests, this speeds up crawling and puts less stress on the host.

Conclusion

Data is often the core of application functionality and web-crawling is a great tool to easily take advantage of public data available online.

Crawling is a diverse, multi-step process with a lot of viable approaches but to start off sticking with syncrhonious requests and html parsers like parsel can be more than enough for most projects.

For further reading it's important to take a look at web caching and rate limiting, proxies and failure and memory managing. Finally there's a whole other problem of scaling both crawling and data storage when it comes to millions of results.
I'll be covering these in later blogs

How to fix redis busy

2018-08-28T00:00:00+02:00

Imagine scenario where you have a single redis server (which is single threaded) and you have multiple clients:

Eventhough redis is single-threaded it has no problem handling this setup since it's blazing fast and knows how to manage it's task queue.
However if we ask redis server to perform a lua script evaluation it might take a while, which ends up with clients getting this nasty error:

redis.exceptions.ResponseError: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE

The Fix

Thi fix this we need to apply a fix to the client side - make it wait and retry this ResponseError:

To do this lets patch Redis class in python's redis-py client:

from redis import Redis

class MyRedis(Redis):
    lua_retry_time = 120  # how many times to retry

    # override execute to retry busy errors 
    def execute_command(self, *args, **options):
        if not self.lua_retry_time:
            return super().execute_command(*args, **options)
        wait_time = 0
        while wait_time < self.lua_retry_time:
            try:
                return super().execute_command(*args, **options)
            except ResponseError as e:
                if 'busy redis is busy' not in ''.join(e.args).lower():
                    raise e
                if wait_time == 0:  # only print once
                    print('Redis is busy, waiting up to 120 seconds...')
                time.sleep(2)
                wait_time += 2
        return super().execute_command(*args, **options)

Since every command in this client goes through execute_command method we can override it to keep retrying Redis BUSY errors for a fixed amount of time.

With this patch Redis client will retry the connection every 2 seconds if redis is busy evaluating LUA.

Conclusion

I've been trying to find some in-built solution for this and couldn't believe there's no official approach, neither with redis itself nor python's redis client.
The internet resources are very scarce around this issue too, so I figured I'd try and remedy this.

Quest Accepted: Travel to HCMC, Vietnam

2018-01-18T00:00:00+01:00

Me and my girlfriend are spending 3 months in Vietnam this year (amount tourist visa allows you) and our first stop is the biggest city in Vietnam, and the biggest city we've been to in general!

Quest #1: Getting There

Our original plan was to go fly out from Tartu to Helsinki, chill there for 8 hours (literally) then take an 8 hour flight to Bangkok, chill there for two hours (literally too, air conditioning is strong in asia) and finally take a two hour flight to HCMC.

It didn't go exactly as planned.

Before we left to Helsinki we did a final checkup and turns out our flight carrier Jetstar does not offer online check-in for our trip - this means we would have to leave the international area of the airport, check in, get back to the international area, and get to our gate for the flight. To leave the international area you need to get a visa and we only had 2 hours there so there would have been almost no way for us to make it... Well so we thought. Even Jetstar support "specialist" assured us that is the case:

"you need to get out to check in, would you like to reschedule for an extra fee?".

Nevertheless we found it a bit hard to believe and went to reddit, asked some people who had done these kinds of trips before on /r/digitalnomads and turns out big airports like the one in Bangkok have check-in booths in the international areas so we didn't have worry about anything! Crisis Averted!

Well we weren't too certain, but we were at least somewhat relieved.

However our adventure was not over.
Once we got to Helsinki we found out that everything is really bloody expensive! We chilled in a mall near the airport for 8 hours trying to not go bankrupt and it was a long, dull experience. Also I got a bit of a fever so that definitely didn't make it any more exciting.
Finally we decided to go to the airport early, which was a great idea as the buses in Finland don't stop unless you put your hand out and wave at them like your life depends on it.
Needless to say we missed our bus because we were not aware of this great ritual of Finnish people and enjoyed 25 more minutes of lovely Finnish weather.
Waving hands at busses? Pretty barbaric if you ask me.

There's more!
Finally we got onto our plane from Helsinki to Bangkok. It was a nice plane and the flight went pretty smooth, except that we were late. The flight wasn't delayed or anything but we were almost an hour late for whatever reason.

We landed on time to still make it, still had an hour before check in closes, "We did it" so we've thought. Well our plane couldn't park for another 30 minutes as a plane in front of us had to be towed away as it's wheels were broken. So we are finally connected and we're on the very end of the plane and some folks being decided to chill in the corridor of the airplane with their toddlers and their bags. Politely we managed to climb through their barricades while avoiding the stink eyes and we made it out to the airport with 20 minutes to spare!

We had a 10 minute long sprint through the entire airport looking for Jetstar's booth, but it's nowhere to be found. Finally we found a gate with a small placard saying something related to our journey. There's was a lady standing there and she looked confused when we passed her our phones with flight tickets on them. She asks some questions that we couldn't really make out, we nodded and pointed and weirdly enough we got let through!

We had our bags checked and had 10 minutes to spare! We went on looking for jetstar's booth again but the info desk just pointed us to the gate from which our plane was about to leave. Well the people at the gate told us that we messed up, "how did you even get here without a boarding pass and without going through immigration?" they asked. We explained our story briefly and luckily for us they took pity on us and we were issued boarding passes right there at the gate, and had 5 minutes left to boot!
Plan was very calculated to say the least!

The last step of our trip was complete! We finally felt relieved, and this final flight between Bangkok and Ho Chi Minh City was the most enjoyable flight of my life - it felt as we're flying first class!

Quest #2: Get to the apartment

We actually landed early, not that it mattered this time. The airport was pleasant and straight-forward. We followed "visa" arrows and finally arrived at the visa issue area, passed our documents and passports to a very bored looking policeman and sat down to wait for our turn.
Every few minutes you could hear a lady mumble something over the speaker and a bunch of foreigners starting to look confused, trying to decipher whether it was their name mentioned there and whether they should try and shuffle to the window to receive their passports back.
Finally after 20-or-so minutes our time came and we were called to take our documents and pay up.

All things considered this was one of the most pleasant visa experiences I've had, almost as if it was there just to collect a 25 dollar fee ¯\_(ツ)_/¯

We almost made it.
Two more steps to go:

Exchange some money
get to our airbnb place(which was pretty far away.)

Once we left the gated airport area we were greeted by shouts and offers: "Taxi, Taxi!", "Sim SIM", "Exchange exchange!". We kinda needed all of them but were intimidated by the aggressiveness of the sellers.

We looked through the booths by standing far away to avoid the calls and decided on a few. They all offered pretty much the same service. For whatever reason we chose the loud lady at the first booth we saw as she wrote down the biggest number on a paper piece as what she would give us per 1 dollar. We handed her over $500 and she quickly turned us away with a stack of vietnamese dongs as she was about to receive a few more customers.
We were a bit confused and shocked, so we decided to whip out a calculator and see whether we got scammed.
After fiddling around we were surprised that we were $60 short! Instead of 12-something million we received only 10. Needless to say we went back to complain and the lady was pretending to be confused:

"Oh you wanted this rate on the wall?" (as opposed to my imaginary rate).

She quickly went through her computer, printed us a receipt and handed us the money we were missing. This time we counted every single dong to make sure we got everything.

Next we got some sim cards with unlimited data for 150k a pop, it was a pretty good deal, especially compared to european prices. We planned on taking an uber but we got a bit lost on the map and our phone batteries were almost dead. Instead we wrote the address of our place down on a piece of paper before the last electrons evaporated from the lithium batteries and went to a pre-paid taxy booth. It's a booth with some people yelling "Taxi, Taxi" every 30 seconds or so where you pay before you go to prevent being scammed.
The guys at the booth looked at the address paper confused and said they'll get us there for 200k. We didn't have much choice and 7 euros didn't sound that much for a couple of lost and tired Europeans.

We got the paper went outside and were greeted by a guy who took us to our cab. So we started the last bit of our trip. The streets were just packed and guzzling with motorbikes and here we were sitting in a clearly oversized for this region SUV and dragging along through the tiny streets of Saigon.

We didn't mind taking our time though. The car was comfy and the views outside were quite mind-blowing! There was a scooter with two people and a dog just standing on the back of it. There was another one with 4 people stacked on it - two parents and two kids. And another one with chickens on it! And the way the whole traffic worked, or didn't. And the way the sidewalks were covered with fruits, produce and even raw meat, occasionally full dead bodies of chicken, ducks and other animals!
I think we were too tired and jetlagged to have a culture shock, but we definitely felt alien.

Our driver finally stops, whips out google translate and says that our hotel is somewhere here but it's closed. We correct him that it's not a hotel but he still says it should be somewhere here. We looked around and the place looked familiar and some of the houses even looked like the ones in the pictures our airbnb host sent us - we had the address, surely we can find it.
As we were about to leave the vehicle our driver used google translate to ask for a $1 tip "for a drink or food". Being tired and just wanting to get away we decided to humor him and give him that dollar.

He wished us goodbye and we set off. We stumbled around looking for our apartment and finally decided to boot up our laptop and charge our phone off it to get maps going. Turns out we weren't at our destination at all!
Google map directions made out an over 500 meter long zig-zag route that we had to take. That doesn't sound like a lot we thought and went forth. Well it was quite a bit, we went through tiny streets filled with scooters and street dogs that followed us around. 20 minutes later we finally arrived there, at least according to the google maps app. However the place was nowhere to be seen. We tried to call our host but our sims were not capable of calls - only online mobile data. Finally we managed to somehow get in touch with our host and we described where we are and he offered to come pick us up in 15 minutes.

While waiting and walking around we actually managed to find the place! It was a tall white house with a huge gate on a small side-street nearby. We did it, we got greeted by an old lady that let us in to our apartment and we could lay down on the bed and say for sure that we "made it!".

Lessons Learned

In general it wasn't as bad as it could have been. I'd say we got lucky at multiple points through out our trip and somewhat unlucky at others. Nevertheless we made it without any losses or injuries and in hindsight it wasn't bad or difficult at all, just inconvenient.

Tips for future traveling:

Research airport structures. Make sure to get online checkin. Check in as early as possible.
Do not trust money exchanges.
Carry multiple mini-usb cables/power banks for phone charging.
Do not trust taxi services too much.
Finland is weird and expensive.

Why I use GPL license

2017-11-18T00:00:00+01:00

This issue has been discussed to death and I still find people blindly advocating MIT or even WTFPL licenses. I'm far from a lawyer but I feel that there are few very generic and simple details about software licensing people who lead open software projects should understand.

Protect Yourself

First thing first - the license should protect the project contributors and creators. In legal terms it's called no-warranty-provided and exlucsion of it can be dangerous.
Technology law sucks - it's outdated, overly complicated and widely regional where software itself has no borders.

To illustrate tis danger imagine John made a library and used a license that does not fully protect the creators from warranty responsibilities(like WTFPL). Now some company decides to use John's work in their software stack and unfortunately John left a bug in his library that either exposed the company to theft or caused their technology to break. In this case the company could easily start lawsuit against John and argue that it was John's open source library that cause them to go under and John would be liable.

Most linceses however have this included, GPL for example:

For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions.

While some occasional ones like WTFPL don't.

To summarise, whatever license you use make sure to protect yourself - make sure you provide no warranty!

Philosophy and Copyleft

It's important to consider license philosophy as well.

Copyleft is a general method for making a program (or other work) free (in the sense of freedom, not “zero price”), and requiring all modified and extended versions of the program to be free as well.

For example GPL is a copyleft license. It's a "viral" license, which means any forks or modifications (that are published) have to also carry the same license. This protects the projects ecosystem and enables a bunch of benefits for everyone involved.

Benefits

On the other hand it's also worth considering the benefits of viral copyleft nature of GPL type licenses:

Patches - since the license is viral any changes or forks will be public and those changes can be applied to the original project very easily. This huge reason why Linux kernel itself is licensed under GPL.
Freedom - it continues the message of free software. As project lead you have the power to propagate the message as any derivatives will have to carry your choice.

Restrictions

it's too restrictive and puts too many gates in education!

This quote comes up a lot when people argue against GPL, however this argument is false as GPL's restriction only applies to published software
(as per FAQ). So in other words you can modify and learn from software all you want and use it (even for profit) as long as you don't publish it and try to turn it into a product of your own.

Is it too restrictive for business? Maybe, but why would you be concerned about business? The ecosystem should not expect anything from a business, it puts an unnecessary centralization to an unreliable source. At the end of the day a business is out there with one goal in mind - to make money - it's not a bad goal, but putting any expectations for a business to go out of their way and support the ecosystem is wildly unreasonable.

Closing Statement

I'll continue using GPL license for all of my project where possible. It aligns with my ideology of software and I believe I can develop and maintain software where I don't need to rely on adoption.

Additional read:

Copyleft: Pragmactic Idealism
GPL's FAQ
It's possible to sell GPL exception

How to disable thinkpad trackpoint acceleration

2017-11-17T00:00:00+01:00

I've got a new thinkpad laptop and I've been struggling with mouse acceleration on trackpoint, turns out disabling it isn't as straight-forward as you'd expect

To get rid of this abomination of a feature that is enabled by default we need xinput

The Fix

First you need to find the code or name of your trackpoint:

$ xinput list
⎡ Virtual core pointer                          id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPad                id=12   [slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                     id=13   [slave  pointer  (2)]
⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
    ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
    ↳ Power Button                              id=6    [slave  keyboard (3)]
    ↳ Video Bus                                 id=7    [slave  keyboard (3)]
    ↳ Sleep Button                              id=8    [slave  keyboard (3)]
    ↳ Integrated Camera: Integrated C           id=10   [slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard              id=11   [slave  keyboard (3)]
    ↳ ThinkPad Extra Buttons                    id=14   [slave  keyboard (3)]
    ↳ SteelSeries  SteelSeries Arctis 7         id=9    [slave  keyboard (3)]

Our guy is TPPS/2 IBM Trackpoint aka id 13. Next we can see the properties of this prop:

$ xinput list-props 13
Device 'TPPS/2 IBM TrackPoint':
        Device Enabled (140):   1
        Coordinate Transformation Matrix (142): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
        libinput Natural Scrolling Enabled (310):       0
        libinput Natural Scrolling Enabled Default (311):       0
        libinput Left Handed Enabled (312):     0
        libinput Left Handed Enabled Default (313):     0
        libinput Accel Speed (314):     0
        libinput Accel Speed Default (315):     0.000000
        libinput Accel Profiles Available (316):        1, 1
        libinput Accel Profile Enabled (317):   1, 0
        libinput Accel Profile Enabled Default (318):   1, 0
        libinput Scroll Methods Available (319):        0, 0, 1
        libinput Scroll Method Enabled (320):   0, 0, 1
        libinput Scroll Method Enabled Default (321):   0, 0, 1
        libinput Button Scrolling Button (322): 2
        libinput Button Scrolling Button Default (323): 2
        libinput Middle Emulation Enabled (324):        0
        libinput Middle Emulation Enabled Default (325):        0
        libinput Send Events Modes Available (260):     1, 0
        libinput Send Events Mode Enabled (261):        0, 0
        libinput Send Events Mode Enabled Default (262):        0, 0
        Device Node (263):      "/dev/input/event18"
        Device Product ID (264):        2, 10
        libinput Drag Lock Buttons (326):       <no items>
        libinput Horizontal Scroll Enabled (327):       1

To disable acceleration we are interested in one of these two settings:

        libinput Accel Speed (314):     0
        libinput Accel Profile Enabled (317):   1, 0

We can either diminish/change the acceleration speed by:

# to disable we can use negative acceleration
$ xinput set-prop 13 314 -1
# or we can disable acceleration profile entirely
$ xinput set-prop 13 317 0 1

Note, don't use both, -1 with disabled acceleration pretty much disables the devices.

Make It Persisntant

To make the change persist after restart simply add the config line it to your ~/.xsessionrc file, e.g.:

echo "xinput set-prop 13 317 0 1" >> ~/.xsessionrc

Going to Vietnam!

2017-11-14T00:00:00+01:00

Me and my girlfriend have been planning to spend this year somewhere abroad. My first choice was Lisbon, Portugal as we visited Porto last year and I've always wanted to spend a year or two living and experiencing Portugal. However due to huge house market inflation (what was 500€/month two years ago turned out to be almost 1000€ now) it seemed to be a bit pricey.

As an alternative we've started to look into South-East Asia, Vietnam and Thailand in particular. Both of these countries are famous for expats and remote workers. After a bit of research we've decided to go to Vietnam first as visa is easier to acquire and it's both of our first time dealing with visa-required traveling.

Vietnam

Choosing a particular place to stay in Vietnam is a challenge on it's own since it's a long noodle-like country that has surprisingly different environments - from busy, motor-bike filled streets to white sand islands.

Ho Chi Minh City (aka Saigon) is the biggest city in there so we decided to stay there for our first month. It has a large expat community as well as being action packed and busy - the complete opposite of a small Estonian town we're living in right now.

You can get a nice place in the range of 300-500€/month. There are a few places to look for great rent deals:

Airbnb.com - A lot of the places on airbnb offer monthly or weekly discounts. So despite airbnb usually having inflated prices compared to the rest of the market it can still contain a lot of good deals. Also it provides security and allows short term rent.
expat.com - has a lot of listings. The only issue is that browsing is really tedious and difficult since there are no search filters.

On airbnb we managed to snag a 350€ apartment that is close to District 1 which is the city center:

Work

As there are a lot of remote workers from all around the world in HCMC, there are quite a few coworking places to choose from. I found coworker.com and it seems that the monthly prices are in the range of 50-100€. Most of the places offer reasonable scaling for shorter durations, i.e. 1 week is 1/4 of monthly price.

I'm not sure whether I'll go to any of those as I usually just prefer a cafe or a lounge of some sorts, but nevertheless it could be a good opportunity to meet other expats!

Visa

Visa acquisition was as straight-forward as it gets!

Just go to visatovietnam.org, fill some basic data from your passport, uncheck unnecessary "fast-forward" fees and pay up 19$ per person (with additional 25$ stamping fee on arrival) for a 3 month-long single-entry visa. Two days later you'll get a confirmation letter - print out some simple papers, get a photo of your mug and a scan of your passport and you're ready to go!

Flight

The flight prices are very reasonable. The month on November this year turned out to be especially cheap. Our flight is Tartu-Helsinki-Bangkok -> Bangkok-HCMC It's a bit tricky since not many flight searches offer direct flights to HCMC but you can simply organize it yourself, especially if you do not have any luggage since you don't need to organize luggage transfer. We bought tickets directly from Finnair and Jetstar. It's a tiny bit more expensive than buying through various middleman websites but it certainly is safer and easier just to go through the official sources.

For a one-way trip we ended up shelling out:

Tartu-Helsinki-Bangkok - 284€

Bangkok-HCMC - 66€

This is with seat selection near the window. Grand total of 350€ is a pretty great deal for traveling half-way to the other side of the world! And round trip is even cheaper if you're better at planning than I am.

Preparation

Sockets - Vietnam uses different sockets than Europe. There are a few socket-types prevalent but by the looks of it it's mostly Type A sockets. Vietnam also has different voltages of 110 to 220 V at 50Hz compared to european 230V @ 50Hz. So in some rare occasions you'll want a power converter as well. However it's probably easier to get the adapters and/or converters in Vietnam itself.

We aren't really planning on taking anything other than laptops and a camera. There seems to be a decent amount of snatch and pick pocket crimes but we have a secure backpack (Bobby by XDDesign) to keep our valuables safe. Vietnam is known for cheap products, especially when it comes to clothing, so really there's hardly a need to take anything else, it's more fun to go shopping!

Follow Up

This is certainly exciting and I'll be sure to follow up this blog in the near future once we get there on Nov 22nd!

I'll be sure to include spending reports and such as well.

Securing Data With eCryptfs

2017-10-22T00:00:00+02:00

Since I started dabbling in Cryptocurrency I've been feeling a bit paranoid about protecting my wallet information and keys. Fortunately there are already tools that are dead easy to use.
The standard tool for linux is called eCryptfs and let me tell you how awesome and convenient it is!

eCryptfs

eCryptfs is a cryptographic overlay filesystem for Linux. It's a great tool to encrypt some private data like cryptocurrency wallet keys, private pictures — anything really.
It allows you to mount a password protected, encrypted filesystem on your usual unencrypted filesystem:

dex@~/.private $ ls
Access-Your-Private-Data.desktop  README.txt
$ ecryptfs-mount-private 
    Enter your login passphrase:
    Inserted auth tok with sig [xxxxxx] into the user session keyring
    INFO: Your private directory has been mounted.
    INFO: To see this change in your current shell:
      cd /home/dex/.private
dex@~/.private $ cd .
dex@~/.private $ ls 
myetherwallet  btc

Pretty damn cool - simple and secure!

Setup

eCryptfs setup is dead easy!
Since linux version 3.18 eCrypt overlay filesystem is included with core kernel. Simply enable it with:

modprobe ecryptfs

Then we need some tools to easily mount, unmount and generate our filesystem: ecryptfs-utils has everything we would need and is available on every linux package manager:

sudo apt install ecryptfs-utils  # Ubuntu
sudo pacman -S ecryptfs-utils  # Arch
etc. etc.

Once you have it installed you'll find your user-space path populated with a bunch of ecryptfs utils, just type in your terminal ecryptfs- and press tab to see the goodies:

[dex@nanosaurus ~]$ ecryptfs-<tab>
ecryptfs-add-passphrase
ecryptfs-find
ecryptfs-insert-wrapped-passphrase-into-keyring
ecryptfs-manager
...

Finally to initiate an ecryptfs system we need to run:

$ ecryptfs-setup-private --nopwcheck --noautomount

We use --nopwcheck and --noautomount flags here for extra security.

nopwcheck allows us to use different password from our user's password
noautomount disables automatic encrypted system mounting, since it's also not a very great idea and wouldn't work if our encryption password is different from our user's one.

This command will ask you for login and mount passwords. For login password use your own password and for mount password leave it empty as ecryptfs will create it for you using your login password as a seed.

Afterwards the system will be initiated and ~/Private and ~/.ecryptfs directories will be created.

Note: you want to backup your encrypted mount password which is located in ~/.ecryptfs/wrapped-passphrase. Remember that mount password is generated from your login password, so the passphrase is completely useless without your login password - put a copy of this file somewhere where you will never lose it!

Usage

~/Private is your ecryptfs encrypted directory from now on. To use it you must mount it with ecryptfs-mount-private command and once you're done using it use ecryptfs-umount-private command to make it inaccessible once again:

dex@~/Private $ ls
Access-Your-Private-Data.desktop  README.txt
$ ecryptfs-mount-private 
dex@~/Private $ cd .
dex@~/Private $ ls 
myetherwallet  btc diary emberassing_hobbies

That's it! Now you have a safe directory on your computer where you can store sensitive data! Even if someone gets access to your user's homespace they'll still need your ecryptfs password to get anything out of it.

Customizing

~/Private is a pretty terrible name. It's Camelcase and I'd very much prefer it to be hidden.
We can change it to ~/.private very easily though:

mv ~/Private ~/.private
echo /home/dex/.private > ~/.ecryptfs/Private.mnt

General Security tips

The Three Rule

If it doesn't exist in __three__ places it doesn't exist at all.

Fortunately ecryptfs directory acts like a normal directory so you can easily back it up to usb-stick, cloud storage or even email. Really just put that shit everywhere as longs as you have your login and mount passwords safe no one will be able to access your stuff.

We've all heard magical stories about people losing usb sticks with their bitcoin wallets. Well if you had your wallet encrypted and in three different places that would have never happened!

Long Passwords Triumph

I don't think I can top the explanation by this xkcd comic so I'll just leave you with it and say that just have a simple a-z password which is at least 13 characters long.

wisdom.cat

2017-04-01T00:00:00+02:00

I've always wanted to make a webapp and flask always looked appealing but deployment and production hassle really turned me off. A week ago I tried to force myself through it and I have to say, it's really not that bad!

This is not a guide and you most likely won't learn anything - it's more of a condense story of how wisdom.cat was developed and doesn't really serve any other purpose

Wisdom.cat

$ cat wisdom
=( ^ >w< ^ )=
wisdom.cat - is a video aggregator website that aggregates bite size videos
which are in some way educational. The intention of this website is
to have something beneficial to watch during short down-time periods;
https://github.com/Granitosaurus/wisdom.cat

I work remotely and I tend to eat at home so I usually watch a video or two during my lunch break. However I wanted to make sure I don't completely waste this time and don't let my brain fall a sleep or lose track so I've been watching education videos to compensate.
So the goal of wisdom.cat is to solve this issue and provide me and others who are stuck in a similar position something to watch.

Alternatives and Inspirations

For quite a while a brilliant website (that wisdom.cat is more or less a clone of) performed this function - it's called http://unplugthetv.com. However as time went on the feed slowed down to the point where it has been completely dead since October of 2016.
I didn't like few things on that website; first that the feed was fairly slow and the second, much more important one, that there was no way to customize it. So a lot of unwanted things ended up in the feed, my personal example being the TED channel, it used to be brilliant - 4 years ago, now it's utter garbage that if anything makes people more dumb instead of educating them.

So in hindsight you could say wisdom cat was born from a dead cool website and my hatred for TED (TED-ED is really lovely though!).

Some alternatives I came across while doing my research that are worth mentioning:
http://reddit.com/r/mealtimevideos - similar purpose, however it's not focused on education.
Curiousity app - I've never used it but heard one person on irc.#flask say that wisdom.cat is similar.

Databases

I didn't know much about designing webapp data storage at the time and just went with sqlite3, which didn't play well with pythonanywhere hosting that I was trying to use at the time. So I made a switch to mysql, which was a miserable mistake. Mysql is dreadful to work with and I've spent more time working on the database and such than on the web app itself.
Frankly it was extremely unenjoyable experience and since this project was my spare time thing I came to a brilliant conclusion to try something different.

I got Redis! And honestly, it's been a complete pleasure starting the very moment I ran pacman -S redis.
So what's so great about it? Well it might sound clichè but It Just Works™, no really after I installed it via pacman, started the daemon via systemctl and got a python api (python-redis or flask-redis) I had to add pretty much 2 lines to my youtube scraper to store the data and 4 lines to my flask views to display it - it was liberating!

One important thing I learned when it comes to Redis is to "store data the same way you will query it". Since redis doesn't really have a full query mechanism behind it, you might end up stuck without being able to get your data without a bunch of python code to help you out.
My first attempt was to write every video to video_<id> but since I wanted wisdom.cat to have filters for channels that didn't end up being as efficient. Nowhere near of being a huge performance hit but in short time it might become noticeable.
Then I just wrote video data to Redis lists named after channels, i.e. "channel_CGP Grey" will contain a list of videos, that can be updated and modified - which solved this non-issue completely!

Flask

Flask was an easy choice for wisdom.cat; it was popular(read a lot of learning resources), light-weight and in python!
I'm not sure if I have much to add other than few struggles I had as a newb.
The circular imports got me a time or two, because for some reason I decided to keep app object in __init__.py and ended up kinda liking it.

The biggest issue I had was the sessions. Initially wisdom cat would generate video queue and channel subscriptions and put them to flask session, aka secure cookie. Now this brought an impossible to debug issue that made be dippy few times. Turns out the cookie/session ended up being so big it would not update anymore appropriately since cookie has 4k byte limit and the wisdom cat setup, at the time with 30 videos, was already reaching that!
So needless to say it was a bad idea from the very get go, fortunately with redis up and running it was no longer an issue that would be difficult to solve.

All in all it's hard to say something original about Flask since it has such a huge community and so many opinions, research and information already that anything I'd say would be awfully redundant so I'll just say that Flask was a real pleasure to work with and honestly, I can't wait to start a new project already!

Deploying

This was my most dreaded bit. I've been playing with servers to run crawlers and some notification tools for some time but I've never exposed them to public or made them accessible.
Fortunately it all sounds more complicated than it actually is. All you need is an application, wsgi tool, some daemons and an afternoon!

I ended up renting a linode (which is really lovely!) and using arch node with systemd, nginx and gunicorn.
I've been using arch and systemd for few years now so this part was a breeze, however nginx gave me a bit of trouble. Mostly because most of the tutorials, guides and documentation assumed that it's running on Ubuntu which for some reason uses unusual structure and stores servers in sites-enabled and sites-available which were not present in arch(or most of the other distros). Apparently this was to replicated some apache server structure so I decided to join the cool kid's club and replicate the same structure with inludes and system links - it worked out perfectly fine and I with few lines of configs it was working.

Finally I wrote up two daemon services: gunicorn wsgi service to launch and expose the app, and a timer service - to run video crawler every half an hour, to update the video feed.

And with that wisdom.cat was finally drawing it's first breaths!

Conclusion

This was a really fun little project and I highly recommend investing some time into learning how to make a simple web app like this - it opens up so many gates for so many opportunities!

Doom and Stone in Tallinn

2017-01-31T00:00:00+01:00

I'm a big fan of doom metal and to my surprise a brilliant rock club "Woodstock" was holding a concert featuring 3 brilliant bands: Dopelord[PL], Sunnata[PL] and Stoned Jesus[UA].

I've attended Stoned Jesus concert last year at the same place and it was a blast, so even though I wasn't familiar with the other two bands, Stoned Jesus was more than enough to convince me this time.
I tried to keep the other two bands a surprise and only caved during the long and tedious bus ride to the concert itself. I've listened to the top 3 songs on spotify and was completely stoked!

The club Woodstock is pretty small and was totally packed throughout the entirety of the concert, which started at 8:30 and went a bit over midnight, so that's almost 4 hours of hard metal music from three very distinct and brilliant bands!
We had not only Estonian fans, but also Fins, Russians and even Swedes there. Looking back, I'm very surprised how the club managed to fit in so many people.

The band that opened up was Dopelord, a doom/stone metal band from Poland that was founded back in 2010. They're probably the most doom-like band of the three bands and I couldn't help but feel reminded of my favorite band Electric Wizard which just like Dopelord took inspiration from old movies and witchcraft.

The second band was Sunnata, which is also from Poland.
Sunnata formed in 2008 and overall they sound very much doom, though maybe a bit more on the upbeat side, which is not a bad thing by any means, in fact it was very fitting pacing wise for this particular show.

As a side note I'd like to mention the guitarist's wicked-fabulous hair that had a dedicated fan for full potential.

The third and final band was the star of the show - Stoned Jesus. Definitely the most known band in the show and one of the more known recent bands in the stone rock/metal genre. Some people would disagree with me regarding doom metal(or metal in general) classification, but there are definitely noticeable elements in songs like Electric Mistress and it's especially noticeable during live shows!

It was a 5-year anniversary of their most well known album Seven Thunders Roar and the band was very sincere and thankful to the fans for the support throughout the years and to quote the vocalist/guitarist Igor Sydorenko: "5 years ago we played in crappy places like this, but with 10 times less people, so thank you for the support!"
To finish up the show the band played a 20 minute long encore and had a pretty successful stage dive to finish up already exhausted fans!

To summit it - the show was great and I'd like to publicly thank the bands for coming out this far up to Estonia, Tallinn and delivering such a great show! And would highly recommend, given the opportunity, to check these bands out to anyone both live and online.

That's pretty much it, a rather pointless blog but I felt really inspired to share this and publicly thank Dopelord, Sunnata and Stoned Jesus for a great show!

Getting Terminal Size In Python

2017-01-26T00:00:00+01:00

Getting terminal size can be vital for your application, especially if you are doing some serious printing or drawing. However there are some few tricks worth noting that I'd like to share in this blog post.

First of all the recommended way or the "pythonic" way of retrieving terminal size for python3 is:

#!/usr/bin/env python3
import shutil
columns, rows = shutil.get_terminal_size(fallback=(80, 24))

And it works pretty great, well for the most part that is.

The Issue

This particular function is just a high level wrapper around low level cpython function os.get_terminal_size and the only real thing it does is handle an exception and returns fallback values if that's the case.

However there's a huge pitfall with this function and it's that it doesn't work with terminal pipes!
To confirm and test that with we can try this simple script size.py:

import sys
import shutil

columns, rows = shutil.get_terminal_size(fallback=(123, 456))
sys.stdout.write('cols:{}\nrows:{}\n'.format(columns, rows))

$ python size.py
cols:89
rows:22
$ python size.py | cat
cols:123
rows:456

89 to 22 is my actual size however when piping to anything, including python itself, the size seems to fallback to the fallback values, which in most cases defeats the whole purpose of retrieving the terminal size.

Making It Work!

The solution is pretty simple - use the other function instead!
If we use os.get_terminal_size(0) function, we'll get it working with piping too!

To test that lets change our script:

#!/usr/bin/env python3
import sys
import os

columns, rows = os.get_terminal_size(0)
sys.stdout.write('cols:{}\nrows:{}\n'.format(columns, rows))

The thing to note here is the positional argument 0 in os.get_terminal_size() function, which tells which file descriptor to use:

0 - Standard Input
1 - Standard Output
2 - Standard Error

By default both os and shutil functions use 1, which stands for Standard Output. This means that if we pipe and this output detaches itself we get an OsError:

OSError: [Errno 25] Inappropriate ioctl for device

This default somewhat makes sense if the size of your terminal changes when the output is displayed, but I'm having trouble even imagining an example where that would be the case.

So if we were to run our script now, we'd get the results we are looking for:

$ python size.py
cols:89
rows:22
$ python size.py | cat
cols:89
rows:22
# Even if we pipe multiple times!
$ python size.py | grep . | cat
cols:89
rows:22

Edit

As user bearded_unix_guy pointed out on reddit, using stdin(argument 0) might not always work, in particular it wont work when we pipe to our app:

cat - | size.py

In case like above we actually want to use stdout(default argument) since it's not detached. However what about if your app is in the middle:

cat - | size.py | cat

In this case neither stdout nor stdin will work, but sterr(2) will!
So to combine all of these to cover all of the cases we can simply wrap it in a for loop with exception catching:

def get_terminal_size(fallback=(80, 24):
    for i in range(0,3):
        try:
            columns, rows = os.get_terminal_size(i)
        except OSError:
            continue
        break
    else:  # set default if the loop completes which means all failed
        columns, rows = fallback
    return columns, rows

And there you go, you can use that instead of os.get_terminal_size() and have a pipe-foolproof terminal size getter!

Conclusion

I've spent more time than I'd like to admit trying to figure this out and the stackoverflow thread for this subject only left me more confused.
As always if you have any questions, critique or notices feel free to leave a comment!

My Bin: center

2017-01-18T00:00:00+01:00

People are naturally lazy and strive to automate as much as possible. I'm no exception and my user scripts directory ~/bin is full of scripts that make my life easier or at least makes me feel that is.
On how to setup your environment for easily accessible scripting see my Guide: setup for python scripting

Today I want to show you and explain some bits of a little python script for centering text. It's isn't particularly special, but it's a great base to show off how awesome and powerful python command line tools can be!
It's a simple pipeable script that takes in some text, centers it according to your terminal size and outputs it to standard output.My use case for this is for reading poems and by default they are not centered properly. To add to that I also like to constantly resize my terminal window because I'm running i3wm - a tilling windows manager for linux, which forces every window to use up all of the space it can, which makes them very much dynamic and unpredictable.

In other words it turns something like:

$ cat raven.md
Once upon a midnight dreary, while I pondered, weak and weary,
Over many a quaint and curious volume of forgotten lore,
While I nodded, nearly napping, suddenly there came a tapping,
As of some one gently rapping, rapping at my chamber door.
'Tis some visitor,' I muttered, 'tapping at my chamber door-
Only this, and nothing more.'

Into:

$ cat raven.md | center
             Once upon a midnight dreary, while I pondered, weak and weary,              
                Over many a quaint and curious volume of forgotten lore,                 
             While I nodded, nearly napping, suddenly there came a tapping,              
               As of some one gently rapping, rapping at my chamber door.                
              'Tis some visitor,' I muttered, 'tapping at my chamber door-               
                              Only this, and nothing more.'

While taking your current terminal size into account, so no matter what you're doing or where you are the text will nice and pretty.

Source

Now lets take a look at the source code:

    #!/usr/bin/env python3
    import click
    import sys
    import os


    @click.command()
    @click.argument('input', required=False, type=click.File('r'))
    @click.argument('output', required=False, type=click.File('w'))
    @click.option('-l', '--length', help='maximum line length [default:current terminal size]', type=click.INT)
    def cli(input, output, length):
        """Simple, pipeable tool for centering text"""
        columns = os.get_terminal_size(0)[0]  #1
        source = input.readlines() if input else sys.stdin  #2
        _format = '{{:^{}}}\n'.format(length or int(columns))  #3
        for line in source:  #4
            text = _format.format(line.strip())
            if output:
                output.write(text)
            else:
                sys.stdout.write(text)

    if __name__ == "__main__":
        cli()

Explanation

I love click library, which is a tool for creating command line interfaces. It's beautiful, easy and saves so much space and time. So we start off with two positional arguments for input and output filenames, these are not necessary for the pipe logic we need but is a nice addition if there's a need for standalone function and only takes two extra lines, so why not!
Next we have custom option for length which allows overriding maximum line length. In case you have a very huge terminal window and you just want a nice margin instead of the text being at the very center of your screen. Finally there's the program itself:
#1 - We retrieve dimensions of the current terminal window. This returns a tuple of (columns, rows) since we only care about columns we take the first member.
Check out a related blog entry why we are using this function instead of alternatives here
#2 - We decide on which source to use for input, if first position argument is supplied to script, we'll use that as a source, otherwise use standard input.
#3 - This might appear complicated but what we are doing here is creating a format that we will use to format every line of our text. The line evaluates to {:^<terminal_size>}\n now if we call .format() on that we can insert text and it will be centered. For more check out python's string formatting, it's awesome!
#4 - And lastly we have the loop itself. Here we loop through every line, center it and either write it to file if the second positional argument is supplied or put it straight to standard output.

Improvements

You could probably go wild with a bunch of flags and modifications but it's important to remember to KISS - keep it simple stupid. With pipes, aliases and various other shortcuts leaving this script to do one job is very much a good idea! :)

Conclusion

I'd like to encourage anyone who uses ugly awk scripts and aliases just write a short command line application with python and click. It takes no longer than 10 minutes, it's beautiful, usable, readable and easily shareable!
Let me know if you have any questions and stay tuned for more scripts and explanations!

Guide: setup for python scripting

2017-01-17T00:00:00+01:00

Few python scripts can make your daily routines significantly easier and enjoyable. In this post I'll cover how to set up your linux environment to have your scripts always easily accessible.
In other words, how to make this reality:

$ amicool 
Yes you are, keep it up!

To start off we need to create a directory where your scripts be located. A de facto standard is ~/bin:

$ mkdir bin

Handling the PATH

Now depending on your system and the shell you use, this directory might already be created or might not be in your PATH environment at all.
PATH environment variable is a list of locations your shell will look for executable files - programs in other words. So if you're using good ol' bash simply adding this line to your ~/.bashrc file will do:

export PATH="~/bin:$PATH"

Creating the script

Now we can create the script itself:

$ touch ~/bin/amicool

Notice that we do not use .py extension, because linux takes executables in PATH literally and we don't want to type the extension whenever we call the script.
We should populate this script with some actual code:

#!/usr/bin/env python3

if __name__ == '__main__':
    print("Yes you are, keep it up!")

Notice that we start of the file with #!/usr/bin/env python3. This is called shebang, it's basically a header that tells your operating system what to use to execute the file, in this case we want to use python3.
Afterwards we have the most simple of python code that just prints some text when call it directly, you see when you call a python module directly the magic variable __name__ is being set to '__main__' so you can gate some code in your script behind it so it would only be executed if it's called directly as opposed by being imported by some other module.

Making it executable

Lastly we want to mark our file "executable" so the operating system know it can actually call this file as a script or a program:

chmod +x ~/bin/amicool

And there we go! Your script is good to go and whenever you feel inadequate regarding your coolness just type:

$ amicool
Yes you are, keep it up!

How to get scrapy help.

2016-10-30T00:00:00+02:00

Scrapy is a web-scraping framework for python. It's pretty popular and at the moment of writing it has over 16000 stars on github. In terms of codebase scrapy is pretty simple, however there are few things that are not as explicit as they could be in favor of abstraction and development simplicity.
Not to mention millions of websites that provide their own unique scraping challenges.

So if you do end up not understanding something or encountering some of the few scrapy's quirks, how do you go about it?

Stackoverflow Guidelines

First thing you should do is read is how to ask a good question on stackoverflow.
It's a brilliant guide by, without a doubt the biggest Q&A website on the web, and it focuses on how to ask a good questions regardless of the topic. Following these guidelines not only make it easy for people to help you but also easy for you, yourself to formulate your question and understand the issue you are facing!

Where To Get Help?

There are two places you can go to with your scrapy related questions and issues:

Stackoverflow.
The issue with Stackoverflow is that it has a general rule of questions having to be generic, that means asking how to get price on this item on amazon is not a fit question. However the user base on scrapy tag seems to be quite understanding of this and tend to be quite lenient with reports and down-votes, but don't be surprised if your post gets down-voted or put on hold. All you can do is to try and make your issue more generic and hope for the best!
IRC! @ irc.freenode.org #scrapy
Good old IRC has been there for decades and even though it dropped in popularity quite significantly, it's still a great place to get help on any subject and scrapy is not an exception. Feel free to join the channel and ask questions about anything scrapy related; you can find me there too!
Scrapinghub Forums
Scrapinghub is the company behind scrapy and they have a user forum, so naturally it's a great place to look for help when it comes to your scrapy issues!
Reddit
There's an official scrapy subreddit, which isn't very active but I can tell you for a fact that a lot of people that are involved with scrapy keep an eye on it. It's a great place for some discussions that might not fit stackoverflow and irc.

Providing Information

To debug an issue and get the help you need you need to provide information about your problem:

Source Code of your spider, settings.py and pipelines.py files.
Website you are crawling - sometimes people refrain from providing the url in fear of legal issues or some judgment. Don't worry about that, scraping is very much legal and no one will judge you, it might very well be the opposite - people might be more keen to help you scrape some weird porn website than amazon.
Crawl Log (see Producing Logs) - Scrapy logs majority of the events that happen in your spider, so to debug your spider the best resources are these logs.
Spider Output (see Producing Output) - This will rarely be useful for anyone else but yourself, but it can be very useful in some cases.

Once you have these bits you can easily formulate your question and I'm sure someone will help you out!

Producing Logs

To save a log of your spider run you can use UNIX output redirection syntax:

scrapy crawl myspider 2>&1 > mylog.log
# or
scrapy crawl myspider &> mylog.log

Explanation:
1. scrapy crawl myspider - is a scrapy command that will start crawling spider called myspider
2. 2>&1 - is UNIX syntax for redirecting error output to standard output. In UNIX there are types of outputs and in your log you want to have both of them in one file.
3. > mylog.log - is another UNIX output redirection, but this time we redirect the output to file called mylog.log

Tip: points 2 and 3 can be summarized as &> in bash version 4 and up

For logging scrapy uses python's built-in logging module which by itself is pretty awesome! If you look into it, it might appear quite daunting but you can actually just import logging and simply log message to root logger: logging.warning("this page has no next page"). To have simple logging in your spider.

Producing Output

Scrapy can automatically produce output in one these formats:

'xml', 'jsonlines', 'jl', 'json', 'csv', 'pickle', 'marshal'

To do that simply run crawl command with --output flag (-o for short version) and provide a name + file ending of format you want as an argument:

scrapy crawl myspider --output output.json

This will output all items your spider spews out to output.json file.

To get help for readability purposes you probably want to use either json or xml since those are most readable and as described in section below parsing-friendly formats.

Tip: You can actually tell scrapy to produce output to stdout directly by setting output argument to -:

scrapy crawl myspider -t json -o - output.json

Inspecting Output

There few tools to parse json or xml content, similar like you'd use sed or grep in unix. The most popular and widely known is probably jq, which I believe translates to json query.
I personally really dislike that jq uses it's own mini-language as opposed to xpath or css selectors we all know, love and use daily.
So in response to this I made PQ! It uses xpath and css selectors as well as support both json and xml parsing.

To put it shortly, using the tools described above you can find specific values of some fields really easily.
Lets imagine we have a bunch of products that have these fields: name and price. Now for some reason Samsung items have weird pricing and we want to find out whether that's the case every time we update the code.

For example using pq we can navigate the prices of items that have some keywords in their names:

cat output.json | pq "//item[contains(@name,'samsung')]/price/text()"

Will find all items that contain "samsung" in the name and output their price values. If you change up your spider an run this command again you can easily navigate whether the values are changing.

You can combine this with scrapy spider redirection to have everything in one line:

scrapy crawl spider --nolog -t json -o - | pq "//item[contains(@name,'samsung')]/price/text()"

Conclusion

Scrapy is a lovely framework and web-crawling is a tricky subjects with a lot of hidden issues, quirks and complexities. Because of it being rather big subjects and every spider having it's own challenges it might be difficult to find help. However I feel if you follow the steps and ideas described in this blog post you'll have a really good chance at getting some help either on stackoverflow or irc!

Do you have any places where you go to with your scrapy or web-crawling related questions? Did I miss something important? Leave the comment below :)

Pycon PL + Warsaw report

2016-10-23T00:00:00+02:00

Last year during Euro-Python 2015 someone from PyconPL had an interesting lightning talk about PyconPL, and since that moment I was sold. However I missed it by few days that year. However this year I made it and as a bonus I checked out Warsaw, Poland too!

You can check my previous blog post how we got to Warsaw here

Before going to Pycon me and my girlfriend took two days to look around Warsaw, Poland. Throughout the trip we stayed in 3 airbnb places. The first day we were really surprised how pretty and big Warsaw is. To give a little background, before the trip we did a bunch of research of what to do in Warsaw and we really didn't find much so we didn't have high hopes to begin with, thus the surprise.

So without further ado, here is what I (and in some cases we) think of Warsaw!

Food and Prices

The city is beautiful and there are so many restaurants and cafes. The first thing we did was go to Mango Vegan Street Food which serves great hummus as well as other vegan dishes at a surprisingly low price, brilliant little place!

Low price is definitely a recurring theme in Warsaw - everything is surprisingly cheaper than in Estonia or Lithuania.
I've heard of a saying: "Poland is an eastern-european country with western-european prices" which I heavily disagree with, unless western-europe is cheaper than eastern in this context. Everything in general is at least 20% cheaper than in Estonia, sometimes significantly more and quality doesn't suffer because of it. The selection of foods and products in general is huge too - one of the benefits of being in central Europe I guess.

As a vegetarian I was surprised by how many vegetarian places and options there are in Warsaw. The vegan trend definitely feels real, which was both a pleasant surprise and a piece of convenience for myself.

Simple Definition of meat

the flesh of an animal used as food

a type of meat

the part of something (such as a nut) that can be eaten

As a side note for vegetarianism itself - I found that a lot of polish people do not consider fish to be meat, which is extremely weird and nonsensical. I've noticed that it's a quite common misunderstanding across Europe and had a very similar thing happen in Spain during my visit for Europython 2015 when the vegetarian sandwich I ordered contained tuna...

Also worth noting that a vegetarian person who eats fish is actually called pescatarian. It's a rather peculiar sounding term but it's there for a reason.

Rurki

There are few traditional Polish dishes, but as someone who's from eastern europe himself, I didn't find them unique or interesting at all. All except one: Rurki.
Rurki is a waffle roll with sweet cream inside of it. It's pretty simple but boy it's delicious! It's sweet, crunchy and creamy at the same time - a perfect dessert!
The best rurki we've eat were located in Zloty Tarasy shopping mall. At the very bottom floor there is a small sweets kiosk that offers the best rurki in the city!

Few Cool Food Joints

There wasn't that much to do in Warsaw, at least during our trip, so we ate out a lot. Here's a short list of my favourite places, that I highly recommend!

Mango Vegan Street Food

As I've mentioned above the place is great. It's quite cheap and the food is great. You can get daily deals (that last whole day) and some weekdays have discounts on some dishes. The food selection isn't great, mostly it's just flavours and variants of few dishes, but it's enough for few visits a week. A great place for a quick snack or a lengthier sit down!
Personal favourite and highly recommended: Pomegranate Hummus!

Bubbleology

Bubble tea used to be a pretty huge fad across the whole Europe few years ago. However these days it's no longer even a thing in Estonia or Lithuania. Bubble tea is a tea drink with syrup-like bubbles that you can suck in using bigger than usual straw, it's a mix of a dessert and a drink which seems to be the strategy of the biggest cafe brand Starbucks, so we know it works!
There's nothing particularity interesting about this shop other than it's themed as a science lab where bubble tea science happens. They really go all out by even dressing the clerks in lab coats and devising made up bubble tea formulas on the walls.

Regarding the prices - they are somewhat steeper than you'd expect even when Bubble tea in general is considered to be more expensive than it should. It wasn't the best bubble tea I had but it was good nevertheless so if you miss bubble tea or even worse - never tried it - check this place out!

Min's Onigiri

Onigiri is a Japanese rice ball that can be can contain some flavouring in the middle of it and topped of with dry seaweed wrapping. Historically it was invented as food for travelers, so it's a perfect food for tourists!
I really enjoyed this place, it only serves around 8 kinds of different rice balls and only 3 of them were vegetarian friendly, but all 3 of them were delicious. It also serves miso soup which is more or less bullion alternative Japanese use.
The onigiri themselves aren't that big, so you most likely want to take several. They are extremely delicious however and the miso + onigiri combo ended up being a perfect autumn food for tourists!

Regarding the prices - it's really reasonable. One rice ball + miso soup ended up being 10 PLN (2.3 euros) and 7 without the soup.

Tel Aviv Food & Wine

Another vegan place, this time though, it's a quite fancy one. The place itself is pretty close to the center and easy to reach, however it's pretty small so you should avoid going there during peak hours.
The food is very fancy, but personally I wasn't impressed - it was very pretty but didn't have taste or price value to match it. I think I would reserve this place for more fancy occasions rather than a lunch break or a meet up.

Green Caffe Nero

There really isn't much to say about this place. It's a rather regular café, but it's really affordable, has decent food, tea and coffee, and most importantly it's quite cozy - so it's perfect for tourist stop for a tired, cold and worn-out tourist as myself!
Sitting down with a cup of hot tea, almost italian sandwich and writing some code or a blog for half an hour is always a pleasant experience, wherever you are.

You can find them pretty much on every corner, sometimes you can even see another one from sitting inside one yourself :D

Few Cool Places

Before we left to Poland we did some research of what to visit but couldn't find much other than war museums that are present in every European city and honestly - they bore me to death. However once we got there we found few places that are definitely worth visiting and here they are!

One place I don't mention that might be worth visiting is Copernicus Science Center. The reason why I don't think it's worth mentioning that it's always full of kids and unless you spend a long time in warsaw to find one day that isn't - it's not worth the headache.

University Library

Library? But we're on vacation!
Weirdly enough it was probably my favourite place in Warsaw. The main attraction was not the library itself, but what's on top of it! The roof of the library is a beautiful garden with all sorts of growth and shapes that create this unique and cosy atmosphere.
There were very few people and the weather wasn't that bad. It's near the river so it can get a bit windy, so pack up a scarf!

Technology Museum and Palace of Culture and Science

Palace of Culture and Science was gifted to Poland by Russia after WW2. It's a really beautiful building that since in the very center of Warsaw.
Aside of being pretty it also hosts a pretty cool Technology museum inside of it.It contains exhibits ranging from old motorcycles to nintendo games and it's really fun!
Few of the highlights: Space technology exhibit, video games, old vehicles (from bikes to planes)

Pycon PL

We got there in the morning of Thursday, the first day of the conference. My girlfriend attended Pyladies workshop while I checked us in. I don't think she enjoyed it as much as I thought she would, but it certainly piqued her interest in programming, so it wasn't bad!

The hotel is huge - it supports over 1600 guests and takes a bit under 5 minutes just to go from one end of it to the other. The cafeteria was great and served a sizeable selection of dishes for breakfast, lunch and dinner. As a vegetarian I only felt left out for one lunch where everything seemed to contain meat which means I ate a full plate of roasted potatoes - just like home!

The first few talks were half interesting and half boring. I feel like with many of the talks tend to be longer than they actually need to and some people tend to dig into details a bit too much - the talks in this pycon were not an exception.
I think a lot of the presenters miss the point that the talk you are giving should pique the interest of the listener or tell a story rather than try sum up a framework or some technique in 10 slides.
For this reason, I enjoyed the lightning talks the most though that remains to be true for every conference I've been to.

During the lunch time I met a Lithuanian which was really surprising. Not only python isn't huge in Lithuania but when I was purchasing the tickets one of the hosts mentioned that I'd be the first and only person from Lithuania to attend PyconPL. I'm glad that it turned out not being the case because Lithuania really needs more Python! (or anything that is not php for that matter :D)

Meeting Scrapinghubbers

This year scrapinghub wasn't sponsoring the conference thus we didn't have a booth. However there were 3 other scrapinghubbers present in the conference other than myself. We chatted and played board games together and I can firmly say that in the two years that I've spent working in Scrapinghub I haven't met a single dislikeable person there and this conference solidified this experience even more!

Shoutout to Pawel and two Michals from scrapinghub who made this conference even more fun than it could have been!

Board and Retro Games

Every evening board game event has been held in one of the conference rooms and that's probably where the majority of the conference was spent. There was a board game rental company that provided the conference with some board games as well as some people bringing their own. We brought Exploding Kittens, Loot Letter and Coup. Exploding Kittens by far received the most attention, though the other two games weren't far behind too.

We didn't get a chance to play any board games that we haven't touched previously with the exception of Mysterium! Or at least Polish version of it, which is the original version of the game and by some considered to be superior. It's a brilliant coop game where one player is a ghost who was hanged for a murder he did not commit and has gives hints to the other players of who was the real murderer! Some people describe it as reverse Dix It which in a way it is, but in my opinion, much more fun and engaging.

Another cool piece of entertainment was retro computer games, a full room of them! They had various consoles and computers ranging from atari to windows 95 Worms Armageddon machines. There were open and running through the whole day and you could play something in-between the talks or after having lunch. My personal favorite was Donkey Konga - a party, rhythm game for GameCube. It's like guitar hero but instead of a guitars, players use a set of bongo drums which create a hilarious and often silly atmosphere!

Summary

Initially we didn't find much to do when we did research about Warsaw, however once we got there we got pleasantly surprised by the city and what it offered! The city is big and beautiful. The center is structured in nice blocks that are easy to traverse and navigate. Every block being full of various small shops, cafés or restaurants. And even though there were indeed only few experience places we enjoyed them greatly!
Pycon was a blast too! The educational opportunities weren't that great (the talks, workshops etc.) but the board games and leisure activities were more than enough to compensate!

Will definitely consider going next year!
Only regret was not taking more luggage.

How to parse complicated json trees.

2016-10-10T00:00:00+02:00

Often when web-crawling you can find access to website's api which provides direct JSON of a product, however it's not always so easy to find what you need in what could be a multi-layer mess of a json.

In this blog-post I'll cover few tools and ways to deal with really ugly json trees that you probably don't want to iterate through manually using dictionary key indices.
If you don't care about the research you can just skip to the right tool and solving of the real life case sections at the end.

Cause

Often websites, especially the ones that sell various products tend to overcomplicate their apis by stacking everything in one huge json tree that is at least 10 layers deep and is impossible to understand for an outsider or maybe even other developers in the company.

In this case we'll take a look at small examples of http://ah.nl responses and how can we deal with them without spending hours trying to reverse engineer the whole process.

Example info:
Product url: http://www.ah.nl/producten/product/wi166580/maggi-opkikker-rundvlees
Product api response: https://ptpb.pw/aZ_S
If you put this response through some json visual tool like http://jsonviewer.stack.hu/ you'll notice what a huge mess it is:

Multiple layers, multiple elements, list in a dict in a list in a dict and to parse that you'd end up doing something like:

import json
data = json.loads(body)
items = data['_embedded']['lanes'][4]['_embedded']['items']

And that's just half-way through the tree. For example to find the sku you'd have to use something like:

sku = data['_embedded']['lanes'][4]['_embedded']['items'][0]['_embedded']['product']['id']

Now that's with hard-coding of list indices which are very likely to change for every product, so on top of that ugly line above you'd have to use multiple list comprehensions to find the correct list item from the lanes or items lists. This is bad, ugly, unreliable and extremely painful to work with.

Tools to Solve This

There are several ways this can be approaches and let me spoil it for you, majority of them are bad, so we'll start off with those.

To demonstrate these tools better we'll be parsing this simple json:

data = """{
    "one": {
        "two": [{
            "four": {
                "name": "four1_name"
            }
        }, {
            "four": {
                "name": "four2_name"
            }
        }]
    }
}"""

All examples below are also available on iPython notebook if you want to mess around with them yourself here

Wrong: Flattening The Json

At first glance this might appear as an obvious solution - just flatten everything to the first level! However this brings out a huge issue with keys. Because every key has to be unique, when flattening the dictionary you need to merge several keys into one to preserve the tree order. If we were to flatten our data, it would end up looking like this:

data = """{
    "one_two_four1_name": "four1_name",
    "one_two_four2_name": "four2_name",
    }"""

In a way you might think it looks nice, but the truth is that it's really unpredictable and hard to parse in a more complex context, since you can only select individual values. This might be useful for some edge cases where you only need 1 field the json tree is only two or tree levels deep, but otherwise it's not worth bothering with.

Wrong: Jmespath, JSONPath and JSONiq etc.

These few libraries in a way designed specifically to solve this issue. It seems that json is notoriously bad when it comes to this issue, so tools like theses are dime a dozen on github and while they are great, they fall short when in comes to web-crawling or similar use cases.

However there are two major issues with these tools:

First one being that some of them like the very Jmespath's expressions root-bound which means non-rooted expressions like xpath's //product/name are not possible. This means that you need to write this ugly chain which is barely different to our dict key indices one:

root.foo.bar[].foo2.bar2.product.mynode

The only improvement here is that we can do a bit of recursion by calling [] for every list element, saving us a few list comprehension calls. And it definitely looks nicer, doesn't it?
It is still bad though since at any point the tree might change and our crawler will break because we are root bound.

The second issue being is that all of them are extremely bloated, to the point where they not only design their own parsing logic but also design their own syntax.

When you are crawling a website you already have your own parsing tools to parse the html/xml (like lxml or parsel) and anything other would just introduce obvious redundancy and unnecessary complexity.

Almost Right: js2xml

First I'd like to start off with and give a shout out to a great tool called js2xml which maintained by Scrapinghub. It pretty much does what it says - converts javascript code to an xml tree and it's almost the right tool for our issue, almost.
Since json is part of javascript, this means we can use this tool to parse it:

from lxml import etree
# we need to wrap our data json in variable declaration
# for js2xml to interpret it
parsed = js2xml.parse('var foo = ' + data)
print(etree.tostring(parsed, pretty_print=True))

This is the result:

<program>
  <var name="foo">
    <object>
      <property name="one">
        <object>
          <property name="two">
            <array>
              <object>
                <property name="four">
                  <object>
                    <property name="name">
                      <string>four1_name</string>
                    </property>
                  </object>
                </property>
              </object>
              <object>
                <property name="four">
                  <object>
                    <property name="name">
                      <string>four2_name</string>
                    </property>
                  </object>
                </property>
              </object>
            </array>
          </property>
        </object>
      </property>
    </object>
  </var>
</program>

As you can see it works and could probably be parsed with xpath. It's really ugly and if we were to write an xpath for it, it would be unnecessary complicated and long, but it would work!
If you are already using it to parse javascript somewhere you might just go with it to reduce dependencies if you wish so.

Right: Converting json to xml and Parsing It With xpath

I found two tools and either one of them combined with either lxml or parsel selectors create this beautiful, perfect json-crawling combo for your crawler!

For unaware lxml is a really great tool for parsing xml and html while parsel is built on top of it to make it even greater, so I highly recommend checking it out! Fun fact - it's also used by scrapy and that's where it originated.

Getting back to the point, the two tools that are pretty much alternative to each other are dicttoxml and dict2xml. They are essentially the same thing but I thought I'd mention both since I'm not sure which one is better and requires the recognition.
For sake of being brief I'll show off dicttoxml + parsel only:

from dicttoxml import dicttoxml
root = dicttoxml(json.loads(data), attr_type=False)
# the tree we get:
<root>
  <one>
    <two>
      <item>
        <four>
          <name>four1_name</name>
        </four>
      </item>
      <item>
        <four>
          <name>four2_name</name>
        </four>
      </item>
    </two>
  </one>
</root>

Now we can parse this tree using parsel.Selector and xpath:

from parsel import Selector
sel = Selector(text=root.decode('utf-8'))
# and get the names with
sel.xpath("//name/text()").extract()
# ['four1_name', 'four2_name']

Pretty mind blowing how we solved this mess with one 300 loc big package from pypi and one short xpath.

Solving Our Example

Now that we have chosen a tool let's see how well it works on a real life example we got ourselves at the beginning of this blog: http://www.ah.nl/producten/product/wi166580/maggi-opkikker-rundvlees

I'm going to spoil you the joy of reverse engineering the products api and tell you the api url in this case is: 'http://www.ah.nl/service/rest/delegate?url=/producten/product/wi166580/x'

Lets assume we already have the page source in body variable and dive in:

from dicttoxml import dicttoxml
from parsel import Selector
import json

data = json.loads(body)
sel = Selector(text=dicttoxml(data, attr_type=False))
# now we can find things very easily!
# sku:
sel.xpath("//product/id/text()").extract()
# [u'wi166580']
# price:
sel.xpath("//product//pricelabel/now/text()").extract()
# [u'0.82']

Conclusion

Mission accomplished! We managed to parse multi-layer monster with very few, simple xpaths and a small package from pipy!
Personally I wish I started doing this earlier because iterating through monsters like this one key at the time is extremely tedious and it breaks every time the website decides to update something.
Hopefully this write up can save someone few hours and an early balding. :D

Going to Pycon Poland!

2016-10-10T00:00:00+02:00

Last year in Euro-Python 2015 someone from PyconPL had a lightning talk about PyconPL and since that moment I was sold. However I missed it by few days. This year however, I'm going and I'm taking a short vacation to explore Warsaw too!

There's a bunch of info about the conference on the official page and I don't want to be redundant, but the venue looks awesome and in overall the event looks pretty huge. Primarily I just wanted to share how much money do you need to attend something like this, how do you get there and lastly whether it is worth it.

The route

So we're taking a bus from Tartu -> Riga which is really the most tedious part of this trip. As you can see in the image the trip almost takes 4 hours and the bus leaves early and often is packed full. We'll spend two hours in Riga's airport waiting for our flight, which I really don't mind since Riga's airport is really nice. And to finish it off we'll take a 1.25 hour long flight to Warsaw itself. I love flying, more accurately I like take offs and landings so having such short flights, where majority of the time will be the take off and the landing, sounds great!
In conclusion, we leave at 7:00 and we should be in Warsaw at 14:35 (warsaw's time). So this ends up being 8.5 hours for 900km which really doesn't have a good hour/km ratio per se but considering the down-times and the location it really isn't bad at all!
Lastly the conference organizes a bus that will take us from Warsaw to Ossa village, which is few kilometers away from Warsaw and there the conference itself will be held.

The Cost

Conference

Since I was a bit late to register and missed the early bird prices, I had to drop 211€ per person (422€ for two people) for 4 days of the conference, including food and accommodation. This might seem like a lot but compared to other conferences is really little.
I actually chatted with one of the hosts a bit and he mentioned that pretty much the whole fee goes to the hotel that is hosting the conference, after visiting the website I can understand that since it advertises the rooms at 90€ a night the conference fee seems to be very reasonable indeed!

Travel

The plane tickets from Riga to Warsaw and back ended up being 62€ per person(125€ for two) which is slightly above from the best I could find. At one point I got 42€ deals pop up but the payment didn't go through and the next day it popped to 62€. I know airline websites are really fishy when it comes to pricing, storing profiles and cookies to jack up the price whenever they see fit but I'm certain this was just an unfortunate coincidence.
The bus to Riga from Tartu ended up being 30€ per person (60€ in total) both ways.

So total ended up being 303€ per person which is pretty cheap for a conference 906km away!
This doesn't include any traveling expenses which I'll be sure to calculated and include in aftermath blogpost!

The last python conference I went to was EuroPython 2015 and it was super fun, mostly because I got to meet a bunch of coworkers from Scrapinghub, I hope PyconPL can live up to the hype and I'll be sure to post about it either way.

First post. Hello Pelican!

2016-10-09T00:00:00+02:00

I've decided to start a blog after Python package called Pelican caught my eye.
Pelican is a tool to generate a static blog from reStructuredText or Markdown input files. And most importantly it looks to be really fun, full python with jinja2 templating, which means it's fully extendable, configurable and modifiable as it's under GPL license.

Installing Pelican

The setup for Pelican is pretty straightforward just run:

~> pip install pelican  # Installing Pelican package for python
~> mkdir blog && cd blog  # Create and jump into your blog directory!
~/blog/> pelican-quickstart
    ... #answer some simple questions here
~/blog/> vim content/first-page.md
    ... #write your blog here in simple markdown
~/blog/> pelican content  # regenerate website
~/blog/> cd output
~/blog/output> python -m pelican.server  # run pelican server to test locally

Now connect to http://localhost:8000 and there you go!
You can check here for how to template your message how to format your blog entry.

Vim markdown highlight for .md files

While going through the installation I've noticed that markdown doesn't have highlighting in vim which was peculiar. I found this post which describes a simple fix.
Simply create directories and file:

~/.vim/ftdetect/markdown.vim

with content:

au BufNewFile,BufRead *.md  setf markdown

Theming Pelican

The default Pelican theme is pretty great however I stumbled on flex-theme on pelican theme repo on github. So that's my choice for now, but I'd like to touch up the color scheme a bit. Check out pelican-themes.

Configuring Pelican

A lot of bells and whistles come straight out of the box with the pelican and your theme. For example to setup Disqus commnets all I had to do is add DISQUS_SITENAME = "granitosaurus" where granitosaurus is my registered name of my disqus account.

Publishing Pelican

Since Pelican generates a static webpage you can use anything to publish it. I decided to use github user pages which is a bit more complicated than the docs make it out to be. For user pages I like to keep the whole source code in branch source and keep the generated output in master as per github's user pages rule. Then use ghp-import to automatically update master code with the most recent

$ git checkout -b source
$ pelican -o output -c publishconf.py
$ ghp-import output -b master -p

The above will make source branch, generate blog and push the output to master so it's viewable at https://username.github.io

NOTE The most important bit is to set SITEURL in your publishconf.py to https://username.github.io make sure it's HTTPS since default SITEURL generated by pelican is http and github pages requires https. This took me an hour of messing around to finally figure out.

Wrap Up

So far Pelican took quite a bit of work to get things going. It looks quite simple but there's a bunch of little quirks that are really hard to debug. It's not as easy as starting up a wordpress blog but it's quite fun and it seems to be really flexible.
Let's see if it pays off!

Checkout the source for more at https://github.com/Granitas/granitas.github.io/tree/source

Blog of Bernardas Ališauskas

Memory is Me

Understanding and Improving Memory

Improving Episodic Memory

Emotion

Drugs

Recall

Monica - Personal Contact Assistant

Photography and Video

Journal

Exercise, Meditation and Fasting

Summary and Further Research

Happy 2020

Get Organized With TaskWarrior!

Intro

Example Workflow

Get Micro Motivated!

Further Reading

1 Year in Chiang Mai, Thailand

Visa And Getting Here

Tourist Visa

Education Visa

Getting My Education Visa

Studying Thai

Lodging

Food

Produce

Weeds/leafs/flowers

Coffee and Tea

The Smog

Leisure Time and Conclusions

Credits and attributions

Diving Into Web-Crawling

What's Web Crawling?

Stages

Crawling

Synchronous

Asynchronous

Parsing

Html

Parsing html

Json

Parsing json

Synchronous Example

Conclusion

How to fix redis busy

The Fix

Conclusion

Quest Accepted: Travel to HCMC, Vietnam

Quest #1: Getting There

Quest #2: Get to the apartment

Lessons Learned

Why I use GPL license

Protect Yourself

Philosophy and Copyleft

Benefits

Restrictions

Closing Statement

How to disable thinkpad trackpoint acceleration

The Fix

Make It Persisntant

Going to Vietnam!

Vietnam

Work

Visa

Flight

Preparation

Follow Up

Securing Data With eCryptfs

eCryptfs

Setup

Usage

Customizing

General Security tips

The Three Rule

Long Passwords Triumph

Further Reading

wisdom.cat

Wisdom.cat

Alternatives and Inspirations