Development Blog some things we do
Hello, this is the developer blog of Questionmark, where
we discuss technical challenges and solutions.
If instead you're interested in using our API, please see the API documentation.
If instead you're interested in using our API, please see the API documentation.
06 Feb 2024 by wvengen
Making sense of Nuxt data
To be able to quantify how supermarkets help society to eat healthily and sustainably, we need to know what is on the shelves. Sometimes we need to go into the shops and look at the physical products, but more often we can collect data online. In the past, all websites were server-rendered HTML only, with interactions being handle on the server side as well. Nowadays, most web applications are client-side Javascript applications, providing more direct interaction. Still, to be able...
read more »
14 Nov 2023 by wvengen
Direct previews with Active Storage
Nowadays Ruby on Rails includes Active Storage for handling file uploads, including support for direct uploads from the browser. Recently, we’ve worked on an application that uses the Dropzone file upload component (with the help of Stimulus to wire it up). Until recently, direct previews of movies and PDFs turned out to be a little more involved. To understand why, we’ll first look at how Active Storage enables direct file uploads: The user loads a web page, containing a form...
read more »
19 Feb 2021 by wvengen
PostgreSQL FDW and custom domains
We are using PostgreSQL as a database, with much success. Two of our main datastores are one containing raw product data and another processed product data. Usually this separation holds well, but there are times we’d like to lay them side by side. PostgreSQL FDW allows querying external data sources from within the database. We use it to connect from the processed to the raw database. In these cases there is a limited list of products we’d like to inspect,...
read more »
11 Dec 2020 by wvengen
Matching ingredients with linked data
With a slow rise in the awareness of the climate emergency, there is surfacing some commitment to take action on this, also by governments. One action is to reduce the use of animal-based food, because it has a large impact on the planet, and we don’t really need it to live (as supported by the Netherlands Nutrition Centre, which lays out the framework for a healthy diet in the country). One recent ambition is to reduce the amount of animal-based...
read more »
07 Aug 2019 by wvengen
Product scanner app
The Questionmark API exposes a portion of our food product database. Now and then somebody approaches us who wants to use it for research, build a prototype, or integrate it into their own systems (like Telesuper). This year, Cell-0 approached us if they could use the Questionmark API for a (non-commercial) prototype. It’s fun to see what others can come up with, and cool to see that a browser can scan barcodes, nowadays, for example with QuaggaJS. Even though performance...
read more »
A vital part of Questionmark’s activities is gathering, interpreting and augmenting food stuff data. By doing so Questionmark is able to gain useful sustainability and health-related insights into the assortments of the largest food retailers in the Netherlands. An important step in this process is classifying products. Knowing a product’s category makes it possible to infer all kinds of important information that might otherwise be unavailable, such as serving size and substance density. Every week thousands of products need to...
read more »
02 Mar 2018 by wvengen
Sustainability rankings
Questionmark provides a ranking on health and sustainability for supermarket products. Until now, we’ve been giving products a health rating in five steps and a sustainability score from 1-10. This is derived from life cycle assessment, social product assessment and an animal methodology based on ingredients, origin, packaging, etc. Now and then, we take a fresh look at how our ranking influences society. Then it’s helpful to look at other initiatives that provide rankings on sustainability topics, to find inspiration,...
read more »
13 Feb 2017 by wvengen
Storing barcodes
In a previous blogpost, we talked about barcodes and EAN-numbers, and how in some cases barcodes with different numbers still reference the same product. The solution for searching there was not very performant, and wouldn’t scale to more than a handful of barcodes to check. This post describes how to store barcodes in the PostgreSQL database in a normalized way, so that searching can be done without slowing down queries. As a bonus, we get a stronger uniqueness constraint. Partial...
read more »
31 Jan 2017 by wvengen
Product categorization with machine learning
Questionmark provides information on sustainability and health of (food) products in the supermarket. Right now, we are able to analyze about 40k products (fully or partially). In past years, a lot of manual work was needed to type over details from photos (like name, brand, nutrients, ingredients), and to link them to concepts we have data for (researched ingredients, specific nutrients, etc.). But to keep information up-to-date, and to be able to add new products when they become available, we’re...
read more »
14 Dec 2016 by wvengen
Finding partial barcodes in SQL
With the Questionmark App you can scan the barcode of a supermarket product, and see how sustainable it is compared to other products. This barcode is generally a 13 or 8 digit international article number (EAN/UPC), which identifies the product. These numbers are handed out by the international GS1 organisation. Barcode: a unique identifier? This number is like a unique identifier for a (consumer) product. Pretty uniquely. But not completely. There are rules for this (also NL-specific ones). These aren’t...
read more »
09 Mar 2016 by wvengen
Multi-level auto eager loading
Active Record is great. Being able to split up large SQL queries into meaningful bits, and traversing relations with ease is invaluable. Yet some discipline remains needed. Using eager loading to avoid N+1 queries, to name something well-known to any Rails developer who has some real-world experience. For smaller projects, this is usually fine: the controller does the preloading, all set. But when it grows, and more and more functionality is moved to concerns and other places, it’s not so...
read more »
15 Feb 2016 by wvengen
Reading CSVs from Excel within Ruby
Again and again I hear someone explain about how they manage data and notice that somewhere spreadsheets play an important role. Very often, that’s Microsoft Excel. From big companies to small organisations like ours. And so it’s surprising that it’s such a hassle to reliably export data for use in other tools. What would be the most logical format to export to? Comma separated values (CSV), with UTF-8 encoding. As field-separator preferably a comma (what’s in a name), but a...
read more »
25 Nov 2015 by wvengen
Finding a Ruby memory leak using time analysis
When developing a program in Ruby, you may sometimes encounter a memory leak. For a while now, Ruby has a facility to gather information about what objects are laying around: ObjectSpace.
read more »
11 Sep 2015 by wvengen
Product sustainability in a Sankey diagram
At Questionmark, we research the sustainability of product in (Dutch) supermarkets. On one side, we communicate that to consumers by showing ratings from 1-10 in our app and website. This needs to be easy enough to understand for people to make quick choices, and provide enough detail to understand what the ratings tell. At the other side, we show suppliers and producers why their products get these ratings - and provide information on how they could make their products more...
read more »
12 Aug 2015 by wvengen
Free-form select2 with existing values
In our backend, we’re heavily relying on select2
for choosing values from a list in forms. Sometimes we want to show a list of
existing values, but still allow the user to add his own. This can easily
be done like this:
https://gist.github.com/be89a2f8bfec7c038084.js?file=select2_freeform.js
In our ingredient edit form it looks like this:
read more »