Development Blog some things we do

Web scraping is a powerful tool for collecting data from websites, but what happens to that data after it’s been scraped? Without a reliable way to preserve the exact state of the pages you collect, important details can be lost over time. Imagine being able to not only scrape data but also store it in a way that lets you revisit a website exactly as it appeared in the past, or share a complete, self-contained snapshot of your scraped data... read more »

To be able to quantify how supermarkets help society to eat healthily and sustainably, we need to know what is on the shelves. Sometimes we need to go into the shops and look at the physical products, but more often we can collect data online. In the past, all websites were server-rendered HTML only, with interactions being handle on the server side as well. Nowadays, most web applications are client-side Javascript applications, providing more direct interaction. Still, to be able... read more »

Nowadays Ruby on Rails includes Active Storage for handling file uploads, including support for direct uploads from the browser. Recently, we’ve worked on an application that uses the Dropzone file upload component (with the help of Stimulus to wire it up). Until recently, direct previews of movies and PDFs turned out to be a little more involved. To understand why, we’ll first look at how Active Storage enables direct file uploads: The user loads a web page, containing a form... read more »

We are using PostgreSQL as a database, with much success. Two of our main datastores are one containing raw product data and another processed product data. Usually this separation holds well, but there are times we’d like to lay them side by side. PostgreSQL FDW allows querying external data sources from within the database. We use it to connect from the processed to the raw database. In these cases there is a limited list of products we’d like to inspect,... read more »

With a slow rise in the awareness of the climate emergency, there is surfacing some commitment to take action on this, also by governments. One action is to reduce the use of animal-based food, because it has a large impact on the planet, and we don’t really need it to live (as supported by the Netherlands Nutrition Centre, which lays out the framework for a healthy diet in the country). One recent ambition is to reduce the amount of animal-based... read more »

The Questionmark API exposes a portion of our food product database. Now and then somebody approaches us who wants to use it for research, build a prototype, or integrate it into their own systems (like Telesuper). This year, Cell-0 approached us if they could use the Questionmark API for a (non-commercial) prototype. It’s fun to see what others can come up with, and cool to see that a browser can scan barcodes, nowadays, for example with QuaggaJS. Even though performance... read more »

A vital part of Questionmark’s activities is gathering, interpreting and augmenting food stuff data. By doing so Questionmark is able to gain useful sustainability and health-related insights into the assortments of the largest food retailers in the Netherlands. An important step in this process is classifying products. Knowing a product’s category makes it possible to infer all kinds of important information that might otherwise be unavailable, such as serving size and substance density. Every week thousands of products need to... read more »

Questionmark provides a ranking on health and sustainability for supermarket products. Until now, we’ve been giving products a health rating in five steps and a sustainability score from 1-10. This is derived from life cycle assessment, social product assessment and an animal methodology based on ingredients, origin, packaging, etc. Now and then, we take a fresh look at how our ranking influences society. Then it’s helpful to look at other initiatives that provide rankings on sustainability topics, to find inspiration,... read more »

In a previous blogpost, we talked about barcodes and EAN-numbers, and how in some cases barcodes with different numbers still reference the same product. The solution for searching there was not very performant, and wouldn’t scale to more than a handful of barcodes to check. This post describes how to store barcodes in the PostgreSQL database in a normalized way, so that searching can be done without slowing down queries. As a bonus, we get a stronger uniqueness constraint. Partial... read more »

Questionmark provides information on sustainability and health of (food) products in the supermarket. Right now, we are able to analyze about 40k products (fully or partially). In past years, a lot of manual work was needed to type over details from photos (like name, brand, nutrients, ingredients), and to link them to concepts we have data for (researched ingredients, specific nutrients, etc.). But to keep information up-to-date, and to be able to add new products when they become available, we’re... read more »

With the Questionmark App you can scan the barcode of a supermarket product, and see how sustainable it is compared to other products. This barcode is generally a 13 or 8 digit international article number (EAN/UPC), which identifies the product. These numbers are handed out by the international GS1 organisation. Barcode: a unique identifier? This number is like a unique identifier for a (consumer) product. Pretty uniquely. But not completely. There are rules for this (also NL-specific ones). These aren’t... read more »

Active Record is great. Being able to split up large SQL queries into meaningful bits, and traversing relations with ease is invaluable. Yet some discipline remains needed. Using eager loading to avoid N+1 queries, to name something well-known to any Rails developer who has some real-world experience. For smaller projects, this is usually fine: the controller does the preloading, all set. But when it grows, and more and more functionality is moved to concerns and other places, it’s not so... read more »

Again and again I hear someone explain about how they manage data and notice that somewhere spreadsheets play an important role. Very often, that’s Microsoft Excel. From big companies to small organisations like ours. And so it’s surprising that it’s such a hassle to reliably export data for use in other tools. What would be the most logical format to export to? Comma separated values (CSV), with UTF-8 encoding. As field-separator preferably a comma (what’s in a name), but a... read more »

When developing a program in Ruby, you may sometimes encounter a memory leak. For a while now, Ruby has a facility to gather information about what objects are laying around: ObjectSpace. read more »

At Questionmark, we research the sustainability of product in (Dutch) supermarkets. On one side, we communicate that to consumers by showing ratings from 1-10 in our app and website. This needs to be easy enough to understand for people to make quick choices, and provide enough detail to understand what the ratings tell. At the other side, we show suppliers and producers why their products get these ratings - and provide information on how they could make their products more... read more »

In our backend, we’re heavily relying on select2 for choosing values from a list in forms. Sometimes we want to show a list of existing values, but still allow the user to add his own. This can easily be done like this: https://gist.github.com/be89a2f8bfec7c038084.js?file=select2_freeform.js In our ingredient edit form it looks like this: read more »