Paul Götze’s tech blog

Testing your JSON API in Ruby with dry-rb

2021-10-25T00:00:00Z

Using dry-schema and dry-validate to keep endpoint tests readable and maintainable

Using MJML in Elixir & Phoenix

2020-08-02T00:00:00Z

How to create responsive HTML emails for your Phoenix app with ease

Experiencing The Stroop Effect with a Ruby CLI

2020-06-24T00:00:00Z

You’ve probably seen these weird lists of color words before. Words whose text color does not match their content. Maybe you’ve even tried to read the text colors out loud quickly and have struggled to do so?

These lists are part of psychological tests to demonstrate the so-called Stroop effect. It is named after the American psychologist John Ridley Stroop, who first published the effect in English in 1935.

Wikipedia tells us:

The Stroop effect is the delay in reaction time between congruent and incongruent stimuli.

But what does that actually mean? 🤔

I built a tiny command line interface with Ruby – just because it was fun and so that you can experience the Stroop effect yourself. You can install and run it with:

gem install stroop
stroop

Congruent? Incongruent?

A Stroop tests consists of a small task, like reading color words. This task can be done with different kind of stimuli, namely: neutral, congruent, and incongruent.

Let’s take a list of color words with a neutral stimulus, which means that all words are written in the same text color:

Neutral Stimulus – All color words are written in a neutral text color (black, or here gray)

Reading all words out loud should not be hard to do.

Next let’s take the same list of color words, but let’s print them in a color that matches the respective text. That’s a congruent stimulus:

Congruent Stimulus – The word content matches the text color.

If you read he words again, it might even be easier than with the neutral stimulus.

Last, let’s print the same word list again, but this time we use another text color than the color that is represented in the text. That’s an incongruent stimulus:

Incongruent Stimulus – The word content and the text color are different.

Reading the colors words might feel a bit slower now. However, the Stroop effect is much more apparent, if we slightly change our task:

Instead of reading out loud the color words, try saying the text color for each word as fast as possible!

With the incongruent stimulus you probably need much longer now and you might even make one or the other mistake.

And that’s the Stroop effect: the delay that appears if we speak the text color out loud for the incongruent word list compared to the congruent word list.

Isn’t it fun – and really exhausting?

🌱 The above word lists were generated using the stroop gem:

stroop neutral
stroop congruent
stroop incongruent

with the seed: 134414671674842647560860440639024210370

(FYI: I found this tiny stroop CLI to also be perfect for creating colorful artsy-fartsy wallpapers & online profile backgrounds 😉)

How to find new maintainers for your open source project

2020-05-14T00:00:00Z

This open source software helps you build a healthy team of co-maintainers so that no project gets left behind. – The story behind how and why I built Adoptoposs.org.

The <details> Of a Dropdown

2020-05-09T00:00:00Z

I would by no means consider myself as an expert when it comes to HTML & CSS. But I thought I just share what I found to work really well for me, when I tried to build a dropdown panel without any JavaScript.

You can build a fully functional dropdown panel by using plain HTML and CSS with no JavaScript involved by leveraging the <details> and <summary> tags. Here’s a minimal example:

See the Pen details & summary – dropdown (minimal example) by Paul Götze (@paulgoetze) on CodePen.

If you want to see more advanced dropdowns built with this approach and want to learn about what’s going on here, then read on…

There is an HTML tag that might count among the little known but most powerful tags in the world of websites: the <details> tag.

It usually appears together with another tag that makes the <details> tag shine: the <summary> tag.

It basically does what it says – it shows you a summary. And you can click on it to see some details. The best feature is, that it comes out of the box with each browser, no JavaScript needed for toggling the details. Just plain old HTML. Here’s what it looks like by default:

See the Pen plain details & summary by Paul Götze (@paulgoetze) on CodePen.

Wow, not sparking too much joy, yet. But we’ll get there.

Let’s Spice It Up 🌶️

With a bit of styling, the plain <details> and <summary> snippet from above can already be used for such exciting things as questions (summary) and answers (details) on your FAQ page.

But let us explore some even more exciting use cases. Let’s build a dropdown menu that might live in the navigation on your web page.

First, we put a container around our <details> snippet which will represent the dropdown element. Then we replace the former summary text with a hamburger menu icon and insert a list as the actual dropdown content. Last, we wrap the whole dropdown with a <nav> tag and give all of that some quick color styles. Et voilà:

See the Pen plain details & summary by Paul Götze (@paulgoetze) on CodePen.

The <details> tag comes with an open attribute and a disclosure widget (▶, ▼) which indicates whether the dropdown is opened and details are visible. Let’s hide this marker, so that the click area looks more like a general button.

In Firefox we can do so by setting list-style: none; for our summary. In other browsers you need to apply display: none; for the summary’s pseudo-class ::-webkit-details-marker.

We’ll also fix the cursor behavior for the <summary> on the fly. So, this is what we end up with after applying these fixes:

See the Pen plain details & summary by Paul Götze (@paulgoetze) on CodePen.

Detaching The Details

This might already serve well as a simple dropdown menu. But we can do much better.

Right now, when opening the details, the content enlarges the details container and therefore also pushes down any other content that lives below our navigation bar. In more complex navigations we might like to have a dropdown panel that is detached from the triggering summary. So, next up, we’ll add some styles to display the actual menu content in a position-wise independent panel.

We can get an independent menu content by adjusting its position property. Hence, we wrap our content into a <div> container and give it an absolute position. With this, the opened menu content is now displayed directly below the summary, which is the opening trigger for our dropdown menu.

In order to define if the panel is opened to the left or the right of the triggering summary, we apply a position: relative; for the dropdown <div> container. You can now customize the dropdown panel’s anchor by applying right: 0; or left: 0; (which is the default) to the summary. When using a relative position, we also need to make sure the dropdown panel has a minimum inline size of its maximum content width – else our menu items will have unwanted line breaks.

By giving the dropdown <div> container an additional display: inline-block; we make sure the dropdown panel only opens when clicking the hamburger menu icon directly.

Similar to before, we also add some list styles to make it already look like a menu and make it more distinct from the navigation and the text content below:

See the Pen details & summary – dropdown III by Paul Götze (@paulgoetze) on CodePen.

Opening and closing the dropdown works nicely and it looks like a navigation menu alright. However, there’s still an issue if we open our menu and decide to not click anything in it but leave for interacting with other content on the page. Then our menu will still be wide open and cover underlying content.

So, we need to figure out how to close the <details> again, whenever we click somewhere else outside the menu area.

We can use a neat little trick to reach this behavior – again without any JavaScript involved. When the <details> container is in the open state, then clicking any <summary> content will hide the menu content. We can leverage this behavior by making sure, that the only area we can click on outside the menu content will always be the <summary> area. So, a click anywhere else on the page would always trigger closing the menu.

Technically this is possible by enlarging the summaries ::before pseudo-class to the full view size. This is done by giving it a fixed position and expanding it to all four view corners (setting top, right, bottom, left to 0). In order to fill the whole screen we also need to set the content of the ::before pseudo-class. By default the details content is displayed with a higher z-index than the related summary content, so we don’t need to care about this. You can still interact with the menu content. The next example applies a transparent background color for the summary::before’s content, so that we can see the area covering the entire screen behind the menu content:

See the Pen details & summary – dropdown IV by Paul Götze (@paulgoetze) on CodePen.

Conclusion

We built a fully functional dropdown menu without any JavaScript by using the <details> and <summary> HTML tags and some CSS styles.

This approach can be used for a multitude of different dropdown panels, including navigation menus, sharing widgets, and all sorts of buttons that open a panel with further details or actions. In fact, GitHub is using a similar approach for their clone button and the branch select panel – which is also where I took inspiration from to kind of reverse-engineer the described approach:

With some minor additions and a couple more CSS styles we can make our example into a shiny dropdown menu, that works just as you would expect a dropdown menu to work:

See the Pen details & summary – dropdown V by Paul Götze (@paulgoetze) on CodePen.

As long as you don’t want to put any more complex interactions into the dropdown, the described approach does not need any JavaScript. However, if you don’t have a page reload after clicking a link in the dropdown panel, you would need to add some JavaScript to toggle the <details>s open state. The same applies for any additional actions from within the dropdown panel that should change its open state.

I hope you learned some useful details about how to build dropdowns. We can give the positive summary that you might not always need JavaScript to build interactive web components. HTML and CSS might have you covered in more cases than you think. For dropdown menus and dropdown panels it certainly does.

Announcing Adoptoposs.org

2020-03-30T00:00:00Z

I am happy to announce adoptoposs.org – an open source app that connects open source software maintainers with people who want to help keep projects and maintainers healthy in the long term.

File upload to Google Cloud Storage using a Flask API (3 of 3)

2019-08-06T00:00:00Z

In part 3 we’ll have a look at how to customize upload directories and allow multiple storages for different attachments.

File upload to Google Cloud Storage using a Flask API (2 of 3)

2019-08-06T00:00:00Z

In part 2 we’ll implement the attachment in the model, the file upload endpoint and some upload tests using an in-memory file storage.

File upload to Google Cloud Storage using a Flask API (1 of 3)

2019-08-06T00:00:00Z

In part 1 we’ll setup a basic Flask app for file uploading to GCS using the filedepot package.

Handling complex JSON Schemas in Python

2017-06-28T00:00:00Z

JSON schemas can get confusing if you have to deal with complex data. We’ll look into how to use references to clean up your schemas.

Testing Your Python API App with JSON Schema

2017-05-31T00:00:00Z

A nice way to test JSON APIs is verifying a request’s response against a JSON Schema. Here’s how you can cleanly test your Python API app by using the jsonschema package and a custom assertion helper.

Building a City Search with Elixir and Python

2017-04-19T00:00:00Z

The other day I was wondering whether there was an easy self-made local alternative to something like the Google Places API, that I could use in a Phoenix app. I wanted to search for a city and wanted to get back the city itself, its state, and the country.

I found the free GeoLite2 city dataset, provided by Maxmind, which I could use to create a city search index.

(In case you directly want to dive into the programmatic materialisation of what I came up with, it is available on Github.)

I did a quick search and stumbled upon the searchex project by @andyl. This actually looked like it was exactly what I was searching for. However, there is very little documentation yet. So, unfortunately, I couldn’t really figure out how to get it working.

Then, while thinking about how to approach this, Whoosh, a Python package that I have used at work, came to my mind. Whoosh is a library for indexing text and searching the index. It is pretty easy to set up and delivers great search results with little effort.

With this in my mind, I was wondering whether there was a way to call Python code from Elixir. After some further research and some articles later I found the Erlang library erlport, which allows you to call Ruby and Python code from Elixir. There is also an Elixir wrapper for it, bearing the sounding name Export. You could also use erlport directly in Elixir, but Export gives you some convenient functions on top and a more Elixir-like feeling.

Setting Up a New mix Project & Python virtualenv

In order to get started with our custom city index, let’s set up a prototype mix project, called elixir_python:

mix new elixir_python

Head to the mix.exs file and add the export dependency:

# mix.exs
# ...
defp deps do
  [{:export, "~> 0.1.0"}]
end

Then install the dependencies with:

mix deps.get

For setting up a Python environment you can use virturalenv to create a local virtual environment. Also keep in mind to activate it after creating:

virtualenv -p python3 venv
source venv/bin/activate

We will use Whoosh, so we need a requirements.txt next to our mix.exs that defines the Python dependencies:

# /requirements.txt

whoosh==2.7.4

Install the requirements with:

pip install -r requirements.txt

Next, we need a directory where our Python code will live. Let’s create a lib/python directory where we will put the *.py files later on. You can really put them wherever you want, you just have to link to the directory when using Export.

In your lib/python directory create a geolite2.py file. This is where we will put the code for our city search index. Next, download the GeoLite2 CSV files from dev.maxmind.com and put the English city locations in the /lib/python/data directory. For our Python requirements we will also need a requirements.txt file in our project’s root directory.

Our Elixir code will live in lib/elixir_python/geolite2.ex.

The overall project structure should now look like this:

└── elixir_python
    ├── config
    ├── lib
    │   ├── elixir_python
    │   │   └── geolite2.ex
    │   ├── python
    │   │   ├── data
    │   │   │   └── GeoLite2-City-Locations-en.csv
    │   │   ├── __init__.py
    │   │   └── geolite2.py
    │   └── elixir_python.ex
    ├── mix.exs
    ├── requirements.txt
    └── …

The Python Part

Our geolite2 Python module will have an API composed of two functions:

# lib/python/geolite2.py

def create_index():
    # We will add code here in some minutes...
    pass


def search(query, count=10):
    # We will add some code here soon...
    pass

The first one creates our search index using the GeoLite2 city CSV file. The second lets us search for cities, states or countries and will pass the results back to Elixir.

Indexing the City Data

For each Whoosh index you can define a certain structure, its schema. The schema defines which data you want to store in the index and which fulltext–or content–you want to run the search on.

Our city schema looks like this:

from whoosh.fields import SchemaClass, TEXT
from whoosh.analysis import NgramWordAnalyzer


class CitySchema(SchemaClass):
    city = TEXT(stored=True)
    state = TEXT(stored=True)
    country = TEXT(stored=True)
    content = TEXT(analyzer=NgramWordAnalyzer(minsize=2), phrase=False)

We want to store the city, the state, and the country. The content field will hold the fulltext to search in, in our case it will be the joined city, state, and country name. This allows us to also search for cities, states or countries and provide multiple query terms to narrow down our results. We use an NgramWordAnalyzer and set the phrase argument to False in order to save some space (see this whoosh recipe for more details).

Before creating the index let’s define our directory names and files we want to use along with some handy functions for building the absolute paths to these files:

# lib/python/geolite2.py

import os

# The base directory where out data lies, relative to this file
DATA_BASE_DIR = 'data'

# The actual city data file
CITY_DATA_FILE = 'GeoLite2-City-Locations-en.csv'

# Our base directory where the index files are stored
INDEX_BASE_DIR = 'index'

# The name of our index
CITY_INDEX = 'city'

def index_path(index_name):
    """ Returns the absolute index path for the given index name """

    index_dir = '{}_index'.format(index_name)
    return os.path.join(current_path(), INDEX_BASE_DIR, index_dir)

def data_file_path(file_name):
    """ Returns the absolute path to the file with the given name """

    return os.path.join(current_path(), DATA_BASE_DIR, file_name)

def current_path():
    """ Returns the absolute directory of this file """

    return os.path.dirname(os.path.abspath(__file__))

Armed with these helpers we can now go ahead and define the actual index creation function. We read the CSV file line by line, and create the schema from it. Some lines in the CSV do not represent cities but states or countries, so we skip these lines, unless there is a value in the city column:

# lib/python/geolite2.py

import csv
import shutil

from whoosh.index import create_in

# ...

def create_index():
    """ Create search index files """

    path = index_path(CITY_INDEX)

    _recreate_path(path)

    index = create_in(path, CitySchema)
    writer = index.writer()

    with open(data_file_path(CITY_DATA_FILE)) as csv_file:
        reader = csv.DictReader(csv_file)

        for row in reader:
            _add_document(data=row, writer=writer)

    writer.commit()


def _recreate_path(path):
    """ Deletes and recreates the given path """

    if os.path.exists(path):
        shutil.rmtree(path)

    os.makedirs(path)


def _add_document(row, writer):
    """ Writes the data to the index """

    city = row.get('city_name')

    if not city:
        return

    state = row.get('subdivision_1_name')
    country = row.get('country_name')
    content = ' '.join([city, state, country])

    writer.add_document(
        city=city,
        state=state,
        country=country,
        content=content
    )

Note that the content ("<city> <state> <country>") is the actual text we analyse and put into the index. The rest of the schema properties is just stored data, which we can access again later on in our results and pass on to our Elixir app.

Searching for Cities

Let’s now implement the function for making a search request:

# lib/python/geolite2.py

from whoosh.qparser import QueryParser
from whoosh.query import Prefix

# ...

def search(query, count=10):
    """ Searches for the given query and returns `count` results """

    index = open_dir(index_path(CITY_INDEX))

    with index.searcher() as searcher:
        parser = QueryParser('content', index.schema, termclass=Prefix)
        parsed_query = parser.parse(query)
        results = searcher.search(parsed_query, limit=count)

        data = [[result['city'],
                 result['state'],
                 result['country']]
                for result in results]

        return data

First, we get the city index that we created with create_index(). Then we build an instance of whoosh’s QueryParser in order to parse our query using our city schema. We use the termclass=Prefix here to only match documents that contain any term that starts with the given query text (see the whoosh.query.Prefix docs). The parsed query is then passed to a searcher which finally runs the search and compiles the results for us. In order to keep it simple we collect the needed data in a list of lists. This will be the data we are going to receive from our Elixir function in a moment.

The Elixir part

Our Elixir API will look pretty much the same as the Python API:

# /lib/elixir_python/geolite2.ex

defmodule ElixirPython.GeoLite2 do
  def create_index do
    # We will add code here in some more minutes...
  end

  def search(query, count \\ 10) do
    # We will add some code here later...
  end
end

Calling Python

To prepare for calling our Python functions from Elixir, add a python_call/3 function to the ElixirPython module. It creates a Python instance for us and runs the Python code we provide it with.

# lib/elixir_python.ex

defmodule ElixirPython do
  use Export.Python

  @python_dir "lib/python" # <-- this is the dir we created before

  def python_call(file, function, args \\ []) do
    {:ok, py} = Python.start(python_path: Path.expand(@python_dir))
    Python.call(py, file, function, args)
  end
end

We make use of Export’s Python module. Python.start/1 returns a tuple including a Python instance. In order to pick up our modules we pass the path to our Python directory as base path. Python.call/4 takes care of calling the given Python function from the respective module file.

Creating the City Index

We use the python_call function we just defined to run the create_index function in Python:

# lib/elixir_python/geolite2.ex

defmodule ElixirPython.GeoLite2 do
  import ElixirPython, only: [python_call: 2]

  @python_module "geolite2"

  def create_index do
    python_call(@python_module, "create_index")
  end

  # ...
end

Searching for Cities

To run a search query we use our python_call function again to call the Python search function we defined. The returned value is a list of lists holding the stored index data. We just loop over it and create Maps from it:

# lib/elixir_python/geolite2.ex

defmodule ElixirPython.GeoLite2 do
  import ElixirPython, only: [python_call: 2, python_call: 3]

  @python_module "geolite2"

  # ...

  def search(query, count \\ 10) do
    results = python_call(@python_module, "search", [query, count])

    for [city, state, country] <- results do
      %{city: "#{city}", state: "#{state}", country: "#{country}"}
    end
  end

Running a Search

And we are done with our hunt for a city search index and we can use it now. Make sure you activated your Python virturalenv, then open up iex and give it a try:

source venv/bin/activate
iex -S mix
iex(1)> ElixirPython.GeoLite2.create_index()

⌛

iex(2)> ElixirPython.GeoLite2.search("Berlin", 3)
[%{city: "Berlin", country: "Germany", state: "Land Berlin"},
 %{city: "Berlingen", country: "Belgium", state: "Flanders"},
 %{city: "Falkenberg", country: "Germany", state: "Land Berlin"}]

Yay. It works!

Let’s try another one:

iex(3)> ElixirPython.GeoLite2.search("San José")
[]

Hm, why is that? We couldn’t find any results, although San José is definitely in the index. It’s because our system does not normalise special characters and accents yet. Let’s do this in a final next step.

Handling Special Characters

On the Elixir side this is easy to do. There is an erlang lib called iconv. Let’s just add it to our mix.exs:

# mix.exs

#...
defp deps do
  [{:export, "~> 0.1.0"},
   {:iconv, "~> 1.0"}]
end

Then install the dependency with:

mix deps.get

Let’s now preprocess the query before we pass it to our Python function:

# lib/elixir_python/geolite2.ex

defmodule ElixirPython.GeoLite2 do
  # ...

  def search(query, count \\ 10) do
    query = clean_text(query)
    # ...
  end

  defp clean_text(text) do
    :iconv.convert("utf-8", "ascii//translit", text)
  end
end

When we rerun our query now we get the wanted city in the results:

iex(4)> ElixirPython.GeoLite2.search("San José")
[%{city: "San José", country: "Costa Rica", state: "Provincia de San Jose"},
# ...
]

We still have some problems with e.g. German cities, like Görlitz, that use Umlauts, so let’s transform them to their ASCII counterparts before creating the index:

# lib/python/geolite2.py

import unicodedata

# ...

def _add_document(row, writer):
    """ Writes the data to the index """

    # ...

    # clean up the content that goes to the index by using _cleanup_text():
    content = _cleanup_text(' '.join([city, state, country]))

    writer.add_document(
        city=city,
        state=state,
        country=country,
        content=content
    )


def _cleanup_text(text):
    """ Removes accents and replaces umlauts """

    replaces = [
        ['ä', 'ae'],
        ['ö', 'oe'],
        ['ü', 'ue'],
        ['Ä', 'Ae'],
        ['Ö', 'Oe'],
        ['Ü', 'Ue'],
        ['ß', 'ss']
    ]

    for original, replacement in replaces:
        text = text.replace(original, replacement)

    text = unicodedata.normalize('NFKD', text)
    text = ''.join([char for char in text if not unicodedata.combining(char)])
    text = text.encode('ascii', 'ignore').decode('ascii')

    return text

If we recreate our index and run a Görlitz query now, we will get some fitting results.

Wrapping Up

We managed to build a small local fulltext index for a city search without too much effort. Our system returns great search results and we allowed to search for cities with or without using special characters.

All in all, it does not scale very well, though. I tried to build another, more sophisticated city index including about 4.4 M cities and villages world-wide and the location coordinates (latitude and longitude) and time zone for each place. If you are interested you can find the script for combining the city data and location data in this gist. It took quite some time to build the index (about 40 minutes on my laptop) and resulted in an index file of 1.3 GB size (compared to ~29 MB for the GeoLite2 index). Although it also worked well and you will get fitting search results, it takes about 10 seconds to finish a single request. This approach would need some additional caching and further optimisation in order to be useful in any kind of way.

So, eventually, I ended up using the Google Places API anyway 😉. But, hey: “Wieder was gelernt.”

Why I Don’t Like Switching the Programming Language

2017-03-19T00:00:00Z

A bit more than 5 months ago I started working at Grammofy, where we use Python a lot. Before, I wrote only a tiny bit of production code in Python. I had a rather hard time starting with it and I came to have a lot of fun with it now. As always, there were people who were congratulating that I finally joined the right side. However, after a couple of months of freely forcing myself to work with a new language, I can say that I don’t like to switch to Python.

Just that you get me right: It isn’t solely Python. I realised that I don’t like switching to any language entirely. Ever. Instead I choose to learn even more languages. And most importantly: to never stick to only a single language.

If you are tempted to do exactly this, may it be because you think you already found your favourite language or because everything else looks not as promising as what you’re using right now – here’s why you better should not settle down.

The Excitement

Six months ago. It was the time were we had this book club at my former work and just finished POODR and the “7 languages in (definitely more than 😉) 7 weeks”-book. And because we were still hungry, we directly started reading a couple of Elixir books afterwards. I had all these new languages in my head, each one touting for my attention. I did mostly Ruby the last years, I was looking into Elixir and a bit into JavaScript, but I had not done too much Python before. So, when I started working in my new job, I was curious to see how I would do, spending most of my time with a – for me rather foreign – language.

Well, it turned out to be a nightmare for the first couple of weeks.

The Pain

If you learn a new programming language, it feels a bit like learning a real language from scratch. You hardly understand the basics, and yet you are asked to build full sentences. You are restarting as a “Junior”, even if your position is not called “Junior”. You love what you’re doing, and you decided a long time ago, that you’ll give a shit on your job title. Still, now you are tempted to beg for being titled “Junior”, just to get even more time to understand things.

But no fear! The good thing is: Learning a new language is just like learning a dialect. You already know a lot about the language from stuff you’ve heard before. Breaking it down, it is just a couple of words, rules, and concepts you don’t know yet. Doesn’t sound too hard to manage, does it?

Nevertheless, my first two weeks were horribly unproductive. Every couple of hours I asked myself why people loved Python. What was so special about it, that I couldn’t achieve the same with Ruby for instance? After my first week of subtle frustration, I decided to get to the bottom of what was so special about it. I searched for Python books and after a while I luckily found “Fluent Python”, that did a great job teaching me the basic principles and specialities of the language. Problem solved.

Not quite. I had a motivation but still no clue and writing code went terribly slow.

I realised that when learning a new programming language, you spend a lot of time thinking about how to do things. If you care (and you should!), you want to write some code that other (more experienced) developers can read and modify without getting a headache.

Of course you know all about clean code, object oriented design, patterns etc. The problem is, you still don’t really know:

how to build your architecture
where to put things
how to name things
how to build certain parts like packages, modules, and classes
what conventions are followed by your team
which of the pythillion different syntaxes you should prefer

Luckily most programming languages come with a community-driven or even in-house style guide. Python has the so-called Python Enhancement Proposal number 8–or shorter PEP08.

Next thing I noticed: Whenever you learn a new language, the first thing you should do is setting up some linting tool for the editor of your choice. It will massively speed you up and leaving you with some really nice aha moments. As painful as a linter sometimes seems if you already know what you’re doing, as helpful it is for starting off. You will learn how things should look like, you will get immediate feedback about what is wrong with your code and it will be right at least syntax-wise and style-wise.

The New

Before I wrote Python there were lots of things I could never understand, like:

Why would one ever use 4 spaces for indentation?
Why would you not put spaces between keyword arguments and default values?
Why that waste of space by putting two blank lines between two functions?

(It absolutely makes sense to me now. I’ll leave it to the kind reader to discover the answers and arguments 😏)

The point is, if you don’t explore other languages, you will never see the different communities and thoughts behind them. Learning another language forces you to think about why other communities are doing some things differently. It forces you to question the approaches you just took to be right, because you knew them long enough and you just have a way with them.

I really like looking into the code of libraries I use. Sometimes it’s needed because of lacking documentation. But even if you have all the information you need, there are some good reasons why you still should read some open source library code. First of all, it helps to judge how well the project is built and maintained. It helps to make a well-informed decision on whether to use it or better just move on. Secondly, it will reveal some patterns how the community likes to structure their code. You will–hopefully–see best practices and–wham-bam–you learned an important piece for your daily work again. Additionally, browsing the code of apps and libraries being built with a certain language will get you an idea about what problems can be solved elegantly using it. You will very soon have a feeling for whether the problem you’re currently working on can be solved easier by using another language.

This brings us to a very important point: You should be open-minded to explore alternative solutions. Sometimes this even means trying another language. I always try to not to be prejudiced by all the rumours out there. Just because someone says: “Nah, don’t use it. It’s bullshit!”, doesn’t mean that you are not allowed to verify it is actual bullshit. Maybe it’s not.

Sometimes there is no way around working with your favourite “worst language in the world”. Who has for instance not dealt with legacy code? If you are forced to work with something you don’t like for the moment, here’s what I’d do:

Find something special about the language. You can’t have fun with something that seems utterly boring and useless to you.

When stumbling upon a new language ask yourself:

Which different concepts are used and why?
What specialities does it have?
What kind of problems does it solve really well?
For which problems should it not be used?

I truly believe that once you found something exciting about a language, all the experienced pain and the too-many-new-things will very soon give you small benefits in a lot of fields. Just keep learning!

The Benefit

Unfortunately, there is no language which is perfect for all kind of problems. Every language was built with the goal to solve a certain problem space. This also means knowing about lots of languages will give you the ability to work in different fields. It allows you to solve problems with the fitting problem solver.

I personally can feel a benefit from working with different languages to that effect that it helps me to dynamically choose what to work on. Programming is still a kind of creative work and you can’t always force you into the flow. In case I’m in the mood for doing some fast prototyping I can choose Ruby. Need to solve some Maths? I’ll probably see whether Python can help here. If I like to try some spacy UI stuff, I can leverage JavaScript. If I have to work on some highly available, distributed [put your favourite 🦄 and 🌈 here] loveliness I’ll maybe go and write some Elixir these days. Personally, it helps me to have a more balanced programming experience all day through.

Besides, learning a new language is just a nice challenge. It keeps your head spinning and demands some patience. Sometimes it requires to be more tolerant towards things you’d approach very differently from what you already know. It gives you loads of valuable insights and when you learn together with someone else or you visit some developer meetups while learning, it will even affect your social surroundings.

To start off, you could go and pick a language you could never really stand. Just keep in mind that you don’t have to switch to it. You should learn about its concepts, the reasons why they exist and what the language is used for. If you like what you learned, look at the community behind, understand their approaches, and maybe build something together with them. If you don’t like what you learned, then ask yourself, why people still use it. It’s certainly not because they are stupid.

The Wrap Up

Phew, that was quite a mix of personal stories and insights.

With these, I feel like not sticking to a single language seems to be a rather good idea. So, the question is: Should I learn all the languages out there? How many languages are enough and how could I possibly learn all this and at the same time still have a social life?

I’m totally aware of that it’s nearly impossible to keep up with all the programming languages you already know, much less to fulfill the “Learn a new language each year!”, that some folks are preaching. I think it’s about something else. It’s about developing a mind-set and skills, that allow you to choose interesting projects, move on if it doesn’t fit anymore, and make a living while having fun solving problems.

It’s like with real languages. You don’t have to know every language perfectly but it might help to know some more in-depth to get along in foreign regions. And if you are missing some bits and pieces you can still wave and make some noise to make your point.

So – back to programming languages – these are my final words:

Do not switch! Rather use the language that suits best. Not switching entirely will enlarge your toolset, keeps you open-minded, and will eventually make you a better developer.

Creating Machine Learning Systems with JRuby

2016-10-03T00:00:00Z

All the different programming languages out there seem to be a better fit for machine learning tasks than Ruby, right? Python has scikit-learn, Java has Weka, and there’s Shogun for machine learning in C++, just to name a few. On the other hand, Ruby has an excellent reputation for fast prototyping.

So, why shouldn’t you prototype machine learning systems with Ruby? Challenge accepted! In this tutorial, we will build a system that can automatically categorize BBC sports articles for you.

Oh, and we’ll do it in Ruby. Well, that’s not entirely true—we will use JRuby and Java’s Weka library via the weka gem.

Preparation

First, install JRuby v10.0.0.1+. Then create an ml_with_jruby directory and put the following Gemfile into it:

# Gemfile

source 'https://rubygems.org'

# use your JRuby version here
ruby '3.4.2', engine: 'jruby', engine_version: '10.0.0.1'

gem 'weka'    # this provides us with the weka lib
gem 'scalpel' # used for text processing

In your JRuby environment, run bundle install to install the gems.

Next, download the free dataset of BBC sport articles and move the unpacked article directories into a ./data/training directory.

Finally, move the last two articles of each sports type into a separate ./data/test directory.

Your project structure should look like this:

└── ml_with_jruby
    ├── data
    │   ├── test
    │   │   ├── athletics
    │   │   ├── cricket
    │   │   ├── football
    │   │   ├── rugby
    │   │   └── tennis
    │   └── training
    │       ├── athletics
    │       ├── cricket
    │       ├── football
    │       ├── rugby
    │       └── tennis
    └── Gemfile

The texts in the test directory will be our test files and will be classified with our trained classifier.

Wait…training, classification, classifier? Lots of terms here. Let’s have a quick look at what they mean.

What is Classification?

Classification means “labeling given data”. An article could, for instance, be labeled as Tennis or Cricket. These labels, Tennis and Cricket, are called classes. The algorithm that chooses the label for one of our articles is called the classifier.

Now, there are different types of classification problems: supervised and unsupervised. The first is often referred to as “Clustering”, where you don’t have any example data and you don’t know beforehand into which classes your algorithm will split your data.

Supervised means we have pre-labeled data, like our labeled articles. We use these labeled articles to train our classifier or in other words: to build a model that can decide on how to categorize new data. After the training, we can pass unlabeled articles to our classifier and it will give us a label for each of them.

This said, the three steps to build a system for supervised classification are:

Creating a training dataset from raw data
Training the classifier with the training dataset
Classifying new data with the trained classifier

Creating the Dataset

Let’s start with compiling our training data.

We need some example data to tell our classifier what different article types look like. Computers are smart, but we can’t expect them to take text and have a good gut feeling of what sports it is about. So, the first step is to transform our raw text into a representation with which our classifier can work. A computer should be good with numbers, so we will use a set of numbers that describe the properties of the text (the so-called features).

We have to find some features that can best divide our data into the different article types. We could calculate, for example, the total length of the text or the number of certain keywords in the text. At this step you can be creative, choosing whatever comes to your mind and makes sense. There are feature combinations that work well together, whereas others reduce the performance of the classifier. Once you have a pool of features, you can use algorithms to select the most valuable features. To keep it simple we won’t cover the feature selection in this tutorial and just use our good sense to select a small set of features.

Extracting Features From the Text

We’ll do the feature extraction in a FeatureExtractor class that takes a piece of text and returns a Hash of properties and their numeric representations. Let’s directly process the given text into paragraphs, sentences (using Scalpel), and words. We will need these soon enough as we fill up our features Hash:

# feature_extractor.rb

require 'scalpel'

class FeatureExtractor
  attr_reader :text, :paragraphs, :sentences, :words

  def initialize(text)
    @text       = text.strip
    @paragraphs = text.split(/\n{2,}/)
    @sentences  = Scalpel.cut(text)
    @words      = text.scan(/[\w'-]+/)
  end

  def features
   {} # to be implemented next :)
  end
end

First, we will add some obvious features: Count the appearance of words describing the sports itself, such as “tennis” in an article about tennis, “cricket” in an article about cricket, etc. (note that e.g. “athlet” counts “athletes” as well as “athletic”, and so on).

class FeatureExtractor
  # ...

  def features
    {
      athletics_hints_count: match_count('athlet'),
      cricket_hints_count:   match_count('cricket'),
      football_hints_count:  match_count('football'),
      rugby_hints_count:     match_count('rugby'),
      tennis_hints_count:    match_count('tennis')
    }
  end

  private

  def match_count(word)
    text.scan(/#{word}/i).count
  end
end

It might be interesting how many proper nouns, like names and teams, appear in the text. So we’ll add a capitalized_words_count feature.

Articles about e.g. tennis and athletics might be more likely to talk about women than, for example, football articles. As such, we’ll cover this in a feature that scans for male and female keywords and says which appear most often. Let’s call it gender_dominance.

Also, add some more generic text features, like text_length, sentence_count, paragraphs_count, and words_per_sentence_average.

When you read through a couple of our training articles, it seems like some people have to say more than others, so let’s count the quotes in the text, too.

You get the idea. Just try to extract some properties that probably can distinguish the content of an article from another.

We will add some additional features that indicate whether it’s more a team or individual sport, by counting the number of hints like pronouns (I, my, me vs. we, our, us) and a number_count feature that might indicate sports where scores or times are important.

With this, we are good for now and we can finish up our extractor class:

class FeatureExtractor
  # ...

  def features
    {
      athletics_hints_count:      match_count('athlet'),
      cricket_hints_count:        match_count('cricket'),
      football_hints_count:       match_count('football'),
      rugby_hints_count:          match_count('rugby'),
      tennis_hints_count:         match_count('tennis'),
      capitalized_words_count:    capitalized_words_count,
      gender_dominance:           gender_dominance,
      text_length:                text.length,
      sentences_count:            sentences.count,
      paragraphs_count:           paragraphs.count,
      words_per_sentence_average: words_per_sentence_average,
      quote_count:                quote_count,
      single_sport_hints_count:   terms_count(%w(I me my)),
      team_sport_hints_count:     terms_count(%w(we us our team)),
      number_count:               number_count
    }
  end

  private

  def match_count(word)
    text.scan(/#{word}/i).count
  end

  def capitalized_words_count
    words.count { |word| word.start_with?(word[0].upcase) }
  end

  def gender_dominance
    terms_count(%w(she her)) > terms_count(%w(he his)) ? 1 : 0
  end

  def terms_count(terms)
    words.count { |word| terms.include?(word.downcase) }
  end

  def words_per_sentence_average
    sentences.count.zero? ? 0 : (words.count / sentences.count)
  end

  def quote_count
    text.scan(/"[^"]+"/).count
  end

  def number_count
    text.scan(/\d+[\.,]\d+|\d+/).count
  end
end

Compiling the Training Dataset

We want to compile a dataset from our text features and save it as a file so that we can load it later on and train our classifier. We could also do it all in memory, but when storing the dataset as a file we can have a look into it and get a better understanding of what’s actually going on in this step. Weka provides a nice way for doing this with its Instances class in the Weka::Core module.

In a separate script, load our training texts and extract our features. Create an Instances object out of them and finally store our dataset on our disk. Before we start with this, let’s create another FileLoader and a Text class that will nicely abstract our file loading and feature extraction from a given file.

The FileLoader will return all text files from the given data directory:

# file_loader.rb

class FileLoader
  attr_reader :data_directory

  def initialize(data_directory)
    @data_directory = File.expand_path("../#{data_directory}", __FILE__)
  end

  def files_for(article_type)
    Dir.glob("#{data_directory}/#{article_type}/*.txt")
  end
end

Our Text class allows us passing a text file and getting its features, by using the FeatureExtractor we created above:

# text.rb

require_relative 'feature_extractor'

class Text
  attr_reader :text

  def initialize(file)
    file_path = File.expand_path(file, __FILE__)

    # There seem to be some invalid UTF-8 characters in the texts,
    # so we remove them here.
    @text = File.read(file_path).encode!('UTF-8', 'UTF-8', invalid: :replace)
  end

  def features
    FeatureExtractor.new(text).features
  end
end

We can now use these classes to write a script for creating the training dataset. Create a new file called create_dataset.rb.

First, create an empty Instances object that represents our training dataset. Add a numeric attribute for each feature and a nominal class attribute. We configure the different article types as possible class values:

# create_dataset.rb

require 'weka'
require_relative 'feature_extractor'
require_relative 'file_loader'
require_relative 'text'

article_types   = %i(athletics cricket football rugby tennis)
attribute_names = FeatureExtractor.new('').features.keys

dataset = Weka::Core::Instances.new.with_attributes do
  attribute_names.each { numeric(_1) }
  nominal(:class, values: article_types, class_attribute: true)
end

# ...

Next, calculate the features for all articles and add them to our dataset:

# ...

def feature_list_for(article_type)
  files = FileLoader.new('data/training').files_for(article_type)

  files.map do |file|
    # Remember that Text#features returns a Hash.
    # We only need the feature values.
    # Since the class value is still missing, we append the
    # article_type as the class value.
    Text.new(file).features.values << article_type
  end
end

article_types.each do |article_type|
  feature_list = feature_list_for(article_type)
  dataset.add_instances(feature_list)
end

# ...

Last, we can save all our calculated features to a file in the /generated directory. Instances allows saving and loading datasets to and from different file formats like CSV, JSON, ARFF, and the less common C.45 file format. Let’s pick ARFF (Attribute-Relation File Format) here, which was especially developed to work with datasets for machine learning tasks and is also nicely legible for humans:

# ...
dataset.to_arff('generated/articles.arff')

Run the script in your terminal to create the training dataset:

$ jruby create_dataset.rb # If you're using RVM, this is just `ruby...`

If you have a quick look into the generated .arff file, you’ll see a header with the customizable relation name and the defined attributes, followed by the actual data rows:

@relation Instances

@attribute athletics_hints_count numeric
@attribute cricket_hints_count numeric
# ...
@attribute class {athletics,cricket,football,rugby,tennis}

@data
1,0,0,0,0,47,1,1237,11,3,19,2,1,0,7,athletics
1,0,0,0,0,46,1,901,7,2,20,0,0,2,5,athletics
# ...

With our training dataset compiled we can now go ahead and train our classifier and classify our test articles.

Training the Classifier

There are loads of different built-in classifiers from which we can choose. We could use Bayes classifiers, Neural Networks, Logistic Regression, Decision Trees, and many more. For simplicity we will use the RandomForest classifier. With RandomForest, we get an easy to configure classifier that is based on Decision Trees and performs well for common problems.

It’s time for loading the training dataset and then training a RandomForest classifier. Let’s do it in a new file called run_classification.rb.

# run_classification.rb

require 'weka'

instances = Weka::Core::Instances.from_arff('generated/articles.arff')
instances.class_attribute = :class

classifier = Weka::Classifiers::Trees::RandomForest.new

# The -I option determines the number of decision trees that are used in each
# learning iteration, the default is 100, we increase it to 200 here to gain a
# better performance.
classifier.use_options('-I 200')

classifier.train_with_instances(instances)

Note that we have to manually set the class attribute after we loaded our dataset. This is necessary because there is no information about the position of our class attribute in our ARFF file (it doesn’t always have to be the last one!).

That was easy enough. Our test articles are already waiting for us!

Classifying Test Articles

We can now use our trained classifier to classify the (let’s pretend) unlabeled articles in our data/test directory.

Before we can pass our test articles to the classifier, we have to extract the same features from them as we did for our training texts. Luckily we can use our FileLoader and Text classes again:

# run_classification.rb

require 'weka'
require_relative 'file_loader' # <= added!
require_relative 'text'        # <= added!

# ...

article_types = %i(athletics cricket football rugby tennis)

def feature_list_for(article_type)
  files = FileLoader.new('data/test').files_for(article_type)

  files.map do |file|
    # Remember again that Text#features returns a Hash.
    # We only need the feature values.
    # The class value is still missing, but this time, we append a "missing"
    # as class value. You can use nil, '?' or Float::NAN.
    Text.new(file).features.values << '?'
  end
end

article_types.each do |article_type|
  feature_list = feature_list_for(article_type)

  feature_list.map do |features|
    label = classifier.classify(features)
    puts "* article about #{article_type} classified as #{label}"
  end
end

Here, we load our test texts and pass their extracted features to the classify method of our classifier. After classifying, print out our predicted classes to the stdout.

Run the script and have look at the output:

$ jruby run_classification.rb

* article about athletics classified as athletics
* article about athletics classified as athletics
* article about cricket classified as cricket
* article about cricket classified as cricket
* article about football classified as football
* article about football classified as football
* article about rugby classified as rugby
* article about rugby classified as rugby
* article about tennis classified as tennis
* article about tennis classified as tennis

Yay. Looks like all our articles got the right label!

This doesn’t mean that our classification system is perfect, though. When training classifiers, their performance can be evaluated by an approach called cross validation. Weka also gives us a cross_validate method for our classifier.

Cross validation splits up the training dataset into N different parts with an equal number of instances. By default, it uses 10 splits. Then it takes 9 subsets to train the classifier and classifies the leftover set. This is done until each subset has been classified after training the classifier with the other 9 subsets. With this procedure, you get an idea of how good your classifier performs because you already know all the labels and can calculate certain measures.

Let’s look at the 10-fold cross validation for our classifier:

evaluation = classifier.cross_validate(folds: 10)
puts evaluation.summary

# Correctly Classified Instances         602               82.8061 %
# Incorrectly Classified Instances       125               17.1939 %
# Kappa statistic                          0.7708
# Mean absolute error                      0.1223
# Root mean squared error                  0.2281
# Relative absolute error                 39.9808 %
# Root relative squared error             58.3231 %
# Coverage of cases (0.95 level)          97.9367 %
# Mean rel. region size (0.95 level)      52.7373 %
# Total Number of Instances              727

In the first two lines we see, that our classifier classified only about 83% of our articles correctly. It’s actually not too bad for our small, contrived feature set. You can expect the performance to improve with a set of carefully selected features. It’s up to you, now—let the hunt for the best features begin!

Conclusion

In this article, we used JRuby to automatically categorize sports articles. We went through three basic steps for building a classification system: extracting features from raw texts, building a training dataset, and training a classifier. With our trained classifier, we classified unlabeled articles.

It looks like Ruby can also be your best friend for machine learning tasks and I really encourage you to check out the Weka framework and play around with it a bit. It’s not only a good exercise but also lets you discover that basic machine learning is actually not rocket science! Give it a try, thanks for reading and let me know how it goes.

You can find the code from this blog post on GitHub:

github.com/paulgoetze/ml_with_jruby

Paul Götze’s tech blog

Testing your JSON API in Ruby with dry-rb

Using MJML in Elixir & Phoenix

Experiencing The Stroop Effect with a Ruby CLI

Congruent? Incongruent?

How to find new maintainers for your open source project

The <details> Of a Dropdown

Let’s Spice It Up 🌶️

Hiding The Disclosure Widget

Detaching The Details

Closing The Menu

Conclusion

Announcing Adoptoposs.org

File upload to Google Cloud Storage using a Flask API (3 of 3)

File upload to Google Cloud Storage using a Flask API (2 of 3)

File upload to Google Cloud Storage using a Flask API (1 of 3)

Handling complex JSON Schemas in Python

Testing Your Python API App with JSON Schema

Building a City Search with Elixir and Python

Setting Up a New mix Project & Python virtualenv

The Python Part

Indexing the City Data

Searching for Cities

The Elixir part

Calling Python

Creating the City Index

Searching for Cities

Running a Search

Handling Special Characters

Wrapping Up

Why I Don’t Like Switching the Programming Language

The Excitement

The Pain

The New

The Benefit

The Wrap Up

Creating Machine Learning Systems with JRuby

Preparation

What is Classification?

Creating the Dataset

Extracting Features From the Text

Compiling the Training Dataset

Training the Classifier

Classifying Test Articles

Conclusion