Szymon Krajewski

My Workflow for Agentic Coding

Mon, 09 Feb 2026 00:00:00 +0000

I’ve experimented with agentic coding for a while and I’ve developed a set of techniques – and a bit of intuition – how to work with agent to deliver actual value without sacrificing my soul as a software engineer. Here’s one of them.

The process I follow consists of 4 steps:

Talk
ADR
Plan
Execute

We’ll discuss each of them in details. For simplicity, let’s assume I’ll add a new functionality to existing system. I use Cursor as agentic IDE, so it may influence the process by its system prompt.

Talk

I start the session by talking to the agent. I describe

what’s the problem,
what’s the desired outcome,
my hypotheses what we need to do to achieve the final functionality.

Then I ask the model to check those hypotheses by reading the existing files and form a sense of relationship between them. If the scope is known and small, I explicitly limit the agent to specific working directory. Otherwise, I intentionally force it to discover the structure, giving only a few keywords as a hint.

Then I talk about potential solutions, the same way I talk to my fellow – but at high level. It helps me organise my thoughts and enrich the context further. After a while, I ask agent for the feedback. Then I propose another solution.

The agent often continues my reasoning by writing example code. When I catch something tricky, e.g. potential performance issues, crossing domain boundaries or just misalignment with the original problem, I point that out, asking for clarification or change.

The talk lasts until I run out of the ideas – or time. For now, I have a pretty solid understanding of the problem space, potential solution space but to gain actual benefit from working with agent I ask about the last thing – the potential options I miss but they would fit as well. I often mention something like “think out of the box” to let the agent takes different perspective.

The agent prepares a list of alternative approaches. Some of them are totally abstract but other ones are worth consideration. So I continue the talking. I challenge a selected approach and I learn something new.

This session lasts from 15 minutes – for small features – to one hour – when working with architecture or infrastructural code. This co-operation with agent not only gives my another perspective but also point out other places potentially affected by the changes I want to introduce.

This talk is rich for useful information – problem, requirements, a few potential solution analysed and verified in the app-specific context. I don’t want to lose it, as this may be relevant for future changes as well. Or, it may be a solid initial context for another discussion. So I ask agent for creating the ADR from the session.

ADR - Architecture Decision Record

If you don’t know what is ADR, it’s a Markdown file sitting in the project repository, representing an significant architectural decision¹ – decisions that are usually difficult to change as project grows.

Should we use RabbitMQ or Kafka?
Classic entity-powered CRUD or Event Sourcing?

Or smaller, domain-specific decisions like

how to store calculations?
how to model specific process?
refactor to X, or just leave it as is?

Typical ADR consist of

Problem and context
Decision Drivers
Considered options, with pros and cons of each
Decision Outcome

After the previous step, the agent has most of the information to form a decent ADR. I try to maintain a consistent decision log, so I ask agent to use the template and fill it with relevant information.

Creating ADR is like transforming sometimes chaotic talk into structured document

I read the Decision Record and – to be honest – I prune most of it. The agent captures the context and problem statement correctly, but some considered options and their pros and cons required cleanup so I remove irrelevant bullet points, example code and other noise. I leave the core – what are the options, the best and worse from each, and brief example if it’s valid for the final decision.

I take the final decision with a bit of justification and I ask agent to update the ADR. We’re ready for the next step.

Plan

To avoid distracting the agent, I start a new conversation and include the newly created ADR. Some models, like Sonnet or Opus 4.5 have smaller context window, so more space is needed to avoid auto-summarisation.

I switch to the plan model and I ask agent to create a detailed plan for implementing the feature from the attached document. Sometimes, I add some details about tests or I point out specific directory but I like to let agent figure it out.

The ADR gives agent the specific instructions what is the desired state and what it should avoid. At this stage, I should not be surprised of the plan it proposes – they should follow the decision.

The plan usually contains specific files to create or modify, more or less detailed description of changes or even the ready-to-use code. Plan, comparing to ADR, it’s more sensitive – it refers to specific files, at specific locations, lines, etc. The plan for me is ephemeral. If I decide to postpone the change, I remove the plan but keep the ADR. ADR is about what and why, the plan is about how and where.

ADR is about what and why, the plan is about how and where

Yes, I read the plan. At this stage I want to be at least 90% sure that agent does what I want. It’s hard to be 100% sure when working with non-deterministic system. I talk with agent about the plan, ask for clarification, or I point it out to do things differently.

Last but not least - I try to close the loop by ensure that agent knows how to test their own changes. Sometimes I even let it commit, but I prefer to manually verify the outcome before.

From time to time I use Cursor feature and spin up two agents in separate work-trees to see which one propose a better plan. I just pick the better and continue as usual.

Execute

I let agent follow the plan while I grab the coffee and watch its during the work. Observing how agent read files and provide changes it’s sometimes breathtaking. But the more I do it this way, the new baseline for normal is set and I’m less surprised and more demanding.

As I said, I shouldn’t be surprised by the outcome. The agent does it job when it follows the plan and it provides the working code according to ADR and plan. Sometimes it modifies the place we never talk about that were somehow related by the type system or to support backward compatibility. Even in well-crafted plan there may be blind spots and it looks interesting when agent can find them anyway. And sometimes not. That’s the trick.

The finalisation is on my side. Quick checklist includes:

reading the code and verify results
committing the work or letting agent commit – thanks to rich context agents provides a detailed commit messages.
prepare the pull request, with the help of agent or not

This is TAPE

Talk, ADR, Plan, and Execution. This four step process gives me a fresh perspective and let me deliver faster. It works well in my current environment with things I’m working on. It surely doesn’t fit in every case, and this is the intuition I developed whether go with it or not.

Is it effective? I don’t know, but for me it’s impressive. I can freely form my thoughts in discussion with agent and it properly captures them, structures and follows. What matters more is the cost. I experimented with the frontier models with thinking capabilities, and some sessions costed at least a few bucks – maybe I just talked too much.

With agents or not, I think documenting your decisions is a good practice. At least those significant ones.

ADR sometimes refer to Any Decision Record, may fit well if you want to store decision not strictly related to the architecture. ↩︎

My first steps into 3D printing

Fri, 30 Jan 2026 00:00:00 +0000

One of my hobbies is tinkering with electronics and cheap microcontrollers from the ESP family. To complete a project, I often needed a case for things I built. One option is to buy a pre-made case, but instead I found it to be a good reason to buy a 3D printer.

But this post is not about the decision process of selecting the printer. It’s about my initial experiences with this device and all those things I discovered on the way. Maybe it can help you answer if you should buy a 3D printer or not.

The beginning

I had literally zero experience with 3D printing. I’ve read about it, I’ve watched some reviews of printers, but I had never touched one before. I’ve explored communities like Printables and MakerWorld, and I couldn’t believe people made all of those things on their devices at home. There was something fascinating about the whole 3D printing technology that I decided to enter this world.

I bought the printer, ran the slicer, and after importing the Benchy model¹, I started my first print, waiting patiently and observing the process. It looked magical. The object just appeared right in front of my eyes and a few meters of plastic turned into something different, pretty, and somehow usable. A few prints later I had a feeling that I was downloading actual things from the Internet.

Learning the process

Each subsequent print was like a test. A different color. Different brands of filament. Different material. Different parameters. Each failure has taught me something new. I learned how small details matter and how impatience can destroy even the best model. Considering how many parameters you can set, what materials you can use, and the fact that each model is different, all of this makes the whole process interesting and engaging.

I think the real adventure starts with designing your own models. Being able to make literally anything you want or need (from plastic of course) gives a sense of freedom. For now I’m just remixing other people’s work, but maybe someday I will actually make something from scratch.

Looking at the designer’s toolbox I guess it’s actually a totally different skill. From my software engineer’s perspective, it’s the difference between using a program others wrote and writing your own customized version. Maybe you will find something that fulfills your needs, but if you know how to do it, the limits are gone – at least those related to skills.

Reality check and usability

Ultimately, after some time, I learned that not every model is worth printing. There are tons of free models, looking insanely good, that have no real purpose, unless the possession is the main goal - all sorts of gadgets, toys or things you throw into the corner days after. On the other hand, I found a few useful projects like Gridfinity, which helped me completely change my approach to organisation. Basically, when you can print anything, you also have to decide what not to print.

Surprisingly, the most common question I get when 3D printing is mentioned is “what do you print?”. It’s hard to answer it precisely, at least in my case. A 3D printer has no actual purpose – it’s a tool. It prints stuff, the same way a standard paper printer prints. The outcome is all that matters.

One of my drawer with little electronics in Gridfinity layout

The truth is, the process takes time. I probably devote more time to my printer than I assumed I needed, but treating it as a hobby definitely helped me to not throw everything away. Curiosity was what drove me forward. Otherwise, I would pause after the 3rd print with no idea what to print next and why. If your goal is only the final print, don’t waste your money and time. Just buy exactly what you need.

Or enjoy the process, like I do.

A little ship, classic Hello World in the 3D printworld ↩︎

When working with agent, close the loop

Thu, 22 Jan 2026 00:00:00 +0000

My first attempts with LLMs started in ChatGPT and were limited to silly debug prompts: questions containing stack trace, error logs and code snippets. The workflow was pure copy-paste back and forth between the chat and editor, and – as you might guess – it didn’t scale.

Then agents showed up. They can execute commands and use tools. However, my first interaction with agents looked similar to the old ChatGPT flow, but instead I pasted everything into the sidebar in my editor – Cursor.

At some point I asked myself: if an agent can run arbitrary command, why can’t it also verify what it did, and fix it automatically when it fails? That insight - something that feels obvious today - was a breakthrough discovery for me a few months ago.

Describing an agent as “LLM using tools in the loop” makes sense. An agent uses whatever it can to do the task. However, I see engineers struggle with closing the loop, like I did in the past. I asked for help and I got it, but there wasn’t a loop – that was a ping-pong.

To close the loop, let agents verify what they’ve done. Let them test, run commands, browse pages, capture output, and read logs. The agent shouldn’t have to ask you for those things. Try to remove yourself from the flow.

Removing yourself doesn’t mean giving back full control. I still manually verify results and – most of the time – I observe what agent does, but I don’t have to manually test the result after every prompt to tell the agent what’s wrong.

Using multiple WiFi networks in ESPHome

Sat, 06 Dec 2025 00:00:00 +0000

My current home network is still far away from what I would consider an ideal setup. Besides the main router, I also have a WiFi extender that creates a separate network. From time to time I change the router location, and many of my wirelessly connected devices require special attention often because I forget to update the credentials to the closest network before the move. Not this time.

ESPHome-powered devices used to be the most problematic case. To change WiFi configuration, you need to provide new credentials, recompile and flash the firmware. You can do this over-the-air, but you only have one chance. If you forget to update the configuration before moving your access point (or if you provide a wrong password), you end up searching for the device to flash it the old-school way – over the wire – like I did many times.

Recently, however, I was smart enough to check if ESPHome supports multiple WiFi networks in its configuration, and the answer is yes, it does. It works exactly as I expected.

You can give ESPHome a number of WiFi networks to connect to. ESPHome will then attempt to connect to the one with the highest signal strength.

In my case I ended up with the definition for my main network and the extended one

wifi:
  networks:
    - ssid: !secret wifi_ssid
      password: !secret wifi_password
    - ssid: !secret wifi_ext_ssid
      password: !secret wifi_ext_password

Since then, I’ve had no issues after moving the network equipment.

I’m still looking for a similar solution for my Raspberry Pis. Currently, I manually change the network configuration before moving the access point – which works as long as I remember to do it in the right order.

How to boot Proxmox from USB using GRUB

Tue, 21 Oct 2025 00:00:00 +0000

The last time I was forced to install a fresh Linux system on my old laptop, I had opportunity to learn how to steer GRUB to boot a system using a bootloader located on a USB drive. This little trick saved me dozens of hours – until I tried to do the same with a Proxmox VE image, where it initially failed.

Eventually, I made it work. Here’s a short instruction – for me and for you.

Prepare USB with Proxmox VE

One of my computer ignores the order of boot devices, so the external USB device has always the lowest priority. I can’t simply insert the bootable CD or USB drive and expect it to start automatically.

If you’re reading this, it’s a non-zero chance you probably have the same problem and you’re looking for solution.

For the record, I used macOS and balenaEtcher to prepare the USB drive. I’m not sure whether it makes a difference, but if your results differ from mine, that might be the reason. I used the Proxmox VE 8.0.

On macOS, you can install it using Homebrew:

brew install --cask balenaetcher

Select the ISO and the target USB drive, then let the software do the rest.

Locate the USB drive and boot system

With the USB drive plugged in, reboot the computer.

Once the GRUB screen appears, press C or ESC to open GRUB console. You’ll see the grub> prompt waiting for your input.

Start by listing available drives:

grub> ls

In my case, the result showed (hd0) and (hd1) with multiple partitions. To inspect one, type:

grub> set root=(hd0,

… and hit TAB to see suggestions.

grub> set root=(hd0,

Possible partitions are:

Partition hd0,gpt1: No known filesystem detected - Partition start at 32KiB [...]
Partition hd0,gpt2: Filesystem type fat, UUID 4238-7745 - Partition start at 302K1B [...]
Partition hd0,gpt3: Filesystem type hfsplus - Label 'PVE' - Last modification time [...]
Partition hd0,gpt4: No known filesystem detected - Partition start at 1534730KiB [...]
Partition hd0,apple1: No known filesystem detected - Partition start at 2KiB [...]
Partition hd0,apple2: No known filesystem detected - Partition start at 32KiB [...]
Partition hd0,apple3: Filesystem type hfsplus - Label 'PVE' - Last modification time [...]
Partition hd0,apple4: No known filesystem detected - Partition start at 1534730KiB [...]

I got a pretty long list of unknown filesystems, but two of them – gpt3 and apple3 – looked promising. I picked gpt3

grub> set root=(hd0,gpt3)

At this point I learned that I couldn’t chainload a bootloader, since there was no EFI path I expected to see. However, I found a /boot directory with two interesting files: linux26 and initrd.img.

To boot the Proxmox, I entered the following commands:

grub> linux /boot/linux26 ro
grub> initrd /boot/initrd.img
grub> boot

The system started to boot, and a few minutes later I got Proxmox VE Installer appeared on the screen.

Proxmox is built for BIOS boot

Here’s what I learned:

The chainloader command is EFI-specific and won’t work here because Proxmox doesn’t include an EFI bootloader. Instead, GRUB has to load the kernel directly – which is exactly what we do with the linux and initrd commands. The final boot hands control to the kernel – and after a short while, the Proxmox VE installer appears.

If you struggle with boot the EFI-based image, check the short instruction from my another article.

Smart Home, My Way

Tue, 30 Sep 2025 00:00:00 +0000

When people ask me about my Smart Home, they often expect something high-tech. Instead, they find an ordinary house that just happens to be observable, controllable, and a little more convenient. The trick? A hidden layer that I can build, break, and rebuild however I want.

What’s the smart home

I don’t really like the term “Smart Home”. I’m not sure the official definition even exists, but most of the ones I’ve found mention “remote control”, “internet connection”, and “automation”. Too broad, too vague.

I prefer to think of the smart home as a home that is:

Observable – I know what’s going on.
Controllable – I can influence it.

But focusing on those two factors makes almost every home “smart”, as long as there’s a human in the loop. People can manually control things when needed.

What if we replace the human by automation, an algorithm, or AI? Then we need another way to access the data and control things – remotely. And that doesn’t necessarily mean via the Internet.

No vendor lock-in

Technology lets us do impressive things, but it has one characteristic I personally can’t ignore – it gets old faster than almost anything else.

That’s why I avoid relying on a single vendor promising to build an entire Smart Home ecosystem. I believe it works for some people and in some cases is the best choice for those who want a controllable home. But I know it won’t work for me. I like tinkering much more than such systems allow, and I want the freedom to use whatever I want – like I did with my heat recovery system.

I set up my entire home using Home Assistant, the open-source smart home platform. I mostly use devices that are cheap, easily upgradeable, and fun to play with. Most of them communicate via Zigbee and Wi-Fi.

A hidden layer

Some modern houses look “smart” from the first glance: ubiquitous displays, touch panels, dimmable lights controlled by quietly listening Google Home speaker in the corner of the living room.

My goal was quite the opposite. I wanted to make a standard-looking house, where the “smart” components are carefully hidden. It’s just a regular home – at the first, second, and even third glance.

I like to think of it as a layer. Something that exists on top of (or under) the things I already have. I can add things to this layer, I can remove them, but most importantly – I can drop everything without leaving visual changes.

Dumb devices are fine

Not everything needs to be connected to the Internet. A smart oven is useless when there’s nothing inside. Also, I don’t leave such appliances running without supervision or at least one person at home.

Smart fridge? I don’t buy it. Smart dishwasher that counts tabs? I always keep a spare pack in the laundry.

Whenever a device asks me for my Wi-Fi password, the first question I ask myself is: what’s the point? Most of the time it brings no real benefit.

It’s sometimes hard to buy appliances without “smart” features (try to buy a non-smart TV). But you don’t need to enable them or, at least, connect them to the Internet.

Exit strategy

Because I’ve chosen the “no vendor lock-in” approach, I lost the possibility of buying any kind of official support. I’m the only maintainer of my smart home layer.

But being the only maintainer doesn’t mean that I’m the only user. My family uses it as well.

There’s nothing more annoying that things that sometimes work. That’s why I design and build critical parts of the home in a specific way. Even if Home Assistant goes down, everything remains operational. Lights and blinds are still controlled using switches, the alarm is unaffected, and smart plugs can be turned on and off using the button on the cover.

What gets lost is only

observability – unless some components are decentralised,
controllability – the remote part.

So turning everything off just degrades the Smart Home to regular Home. And at the end of the day, it even looks like one.

This approach guarantees that even if I’m no longer here to update or fix things, my wife will continue living in the same cozy space – maybe just without a few handy automations.

Hobby, not project

I treat what I built as a hobby. I like tinkering, adding components, removing old ones, testing automations, creating scenes, analysing the data, and so on. Some experiments stay as useful automation, others get disabled after a few tries – but that’s how I like it.

I couldn’t predict every potential use case, but I can easily extend and adjust when needed.

Automation is best when you don’t even notice it. We sometimes do notice – especially when it fails to trigger. That’s the moment when I smile to myself, thinking that there’s always room to improve.

Writing in the era of AI

Sat, 16 Aug 2025 00:00:00 +0000

I started writing before the generative AI became popular. Even a simple article always took me a lot of time to write. Nowadays, authors can produce¹ and publish high-quality content faster than ever before, with almost zero effort. On the other hand, consumers no longer have to search and browse the Internet for the answers – they can generate curated content on almost any topic just for themselves.

If any answer can be instantly generated by an LLM, what’s the point of writing?

Today’s authors are content providers for AI models, which crawl and digest every bit of the Internet. What happens next with this data is rather simple – it’s mixed with other human and AI-generated content to feed the next iteration of models.

As a consumer, I don’t just read to acquire knowledge. I still use RSS as my primary (though not sole) source of information. What have I subscribed to? Mostly blogs. Written by actual people who produce human-crafted content about topics I care about and I find useful.

What I enjoy most are articles with an emotional and personal touch. Problem-solution articles where I can feel the author’s struggle to find an answer. An opinion about a product or tool from someone’s unique perspective.

I also browse some kind of news aggregator, like Hacker News. But instead of rabbit-holing into the pagination of the feed, I look for interesting articles and I try to learn more about their authors. Who are they? What do they care about? Can their other writings be interesting to me? If so, I subscribe and wait.

The content I enjoy the most has a common denominator. There’s a person behind.

Writing in the era of AI-generated content is more important than ever – but the nature of content should evolve to reflect the new reality.

I used to write articles that described ideas, but they were disconnected from who I am – I was invisible in most of the paragraphs. But since AI can generate arbitrary, expert-sounding content, it’s far more valuable to show up as a real person with real opinions and thoughts. It may be less polished – but more human.

When I started the blog, I had the idea that I’d write for myself – to reconnect with my ideas, solidify my knowledge, and maybe help someone else along the way. But over the years, I realized it shouldn’t be just about sharing knowledge – but about sharing how I arrived at it.

The knowledge is already out there – AI can spit out easily. Even before AI, there was plenty of information on the Internet. What’s more valuable is the path to the conclusion – showing the process.

How to connect the dots.

Just write as if no one is watching – and everything will be fine.

Let’s put aside the legacy concerns, such as copyrights and also privacy – but that’s a battle I don’t want to fight right now. ↩︎

Monitoring Heat Recovery Performance with ESPHome and Home Assistant

Sat, 26 Jul 2025 00:00:00 +0000

In my home, I have a mechanical ventilation system with heat recovery powered by Stiebel Eltron V-400. Basically, it’s a large fan with air inflows and outflows ducts. Besides some built-in automation, it’s a pretty dumb device. Without a dedicated gateway (which costs extra money), I can only control it or see its current parameters on the controller, either on the unit or in the hall.

I thought it would be nice to collect some data from this device, both for some kind of analysis and out of curiosity. I run Home Assistant, which can display dedicated graphs and dashboards for such purpose. And, because I recently played with ESP boards, the idea was born pretty quickly.

Parts

For this project I used Dallas DS18B20 temperature sensors and a Wemos D1 with ESP8266 onboard. I picked the Wemos D1 because of 5V output, compact size, and because I had a one spare at this moment.

The Dallas DS18B20 is well-known for its 1-wire protocol, which allows you to connect multiple temperature sensors to a single data line. If you’ve ever wondered about the project where you can use more than one DS18B20 temperature sensor on a single board – here’s one.

I also had a 6-pin LED wire, which I split and used to connect the sensors to the board. For power, I used an old phone charger and a USB-C cable.

Most important parts: Wemos D1 powered by ESP8266 and 5x DS18B20

Plan

The heat recovery unit has four air pipes – two pairs of inflows and outflows. One pair handles air exchange with the outside, while the other circulates air within the house. The airflow setup is as follows:

Schematic diagram of Stiebel Eltron V-400 airflow pipes

I expected these four pipes would carry air at different temperatures. Inside each one, I placed a single DS18B20 sensor. I added an extra one to measure the temperature in the attic, where the ventilation unit is installed.

Schematic diagram of DS18B20 placement within the Stiebel Eltron V-400 airflow pipes

As I mentioned earlier, the DS18B20 uses 1-wire interface, so all sensors can share the same data line. Instead of wiring sensors one into another in a form of chain, I connected them in a star topology, where each sensor has its own dedicated wires, but shares a common data pin.

Preparation

First – the sensors. Soldering the first one was fun, but each subsequent one became a bit more tedious. I soldered each sensor to a wire strip about half a meter long.

Wires prepared to be soldered to DS18B20 sensor

DS18B20 sensor with wires soldered

DS18B20 sensor with soldered wires and secured connections

I wanted to connect everything on the breadboard. Unfortunately, I realised I was still missing the Dupont crimper needed to prepare proper pins for the sensor wires. I ended up using Wago connectors to make all connections.

I connected the data line to pin D13 (GPIO13) on the Wemos D1.

ESP8266 Wemos D1 with the DS18B20 sensors connected using Wago connectors on breadcrumb board

Firmware

I didn’t write the ESP firmware myself. Instead, I decided to use ESPHome which works seamlessly with Home Assistant.

Setting up the ESP with ESPHome was one of the most magical experiences I’ve had. Even though I enjoy digging into programming and writing things from scratch, being able to generate firmware from a YAML file is just wonderful.

Identifying sensors

I started with a basic ESPHome project, configured WiFi connection and set up the one_wire protocol.

one_wire:
  - platform: gpio
    pin: GPIO13

Then, after flashing the firmware, ESP printed the addresses of the connected sensors

[13:47:40][C][gpio.one_wire:021]:   Pin: GPIO13
[13:47:40][C][gpio.one_wire:080]:   Found devices:
[13:47:40][C][gpio.one_wire:082]:     0x48e2a7bc12fa3651 (DS18B20)
[13:47:40][C][gpio.one_wire:082]:     0xc903deab6719b425 (DS18B20)
[13:47:40][C][gpio.one_wire:082]:     0x1f64b28d4ca713ff (DS18B20)
[13:47:40][C][gpio.one_wire:082]:     0xa82fbac2e19d54e7 (DS18B20)
[13:47:40][C][gpio.one_wire:082]:     0x702c19f8ab65e038 (DS18B20)

All of my sensors were visible, but there was one problem. I wanted to name each sensor correctly, but I didn’t know which address was assigned to each one. Fortunately, the sensors are very sensitive and I just registered all of them, assigning the names. Then, I warmed up a specific one and observed which temperature reading increased.

That could have been enough, but I decided to use the different colored wires for inner flow and outer flow temperatures, so I needed to update the names accordingly. I ended up with the list as follows.

sensor:
  - platform: dallas_temp
    id: temp_inner_inflow
    address: 0x48e2a7bc12fa3651
    name: temp_inner_inflow
    update_interval: 60s 
  - platform: dallas_temp
    id: temp_inner_outflow
    address: 0xc903deab6719b425
    name: temp_inner_outflow
    update_interval: 60s
  - platform: dallas_temp
    id: temp_outer_inflow
    address: 0x1f64b28d4ca713ff
    name: temp_outer_inflow
    update_interval: 60s
  - platform: dallas_temp
    address: 0xa82fbac2e19d54e7
    name: temp_attic
    update_interval: 60s
  - platform: dallas_temp
    address: 0x702c19f8ab65e038
    name: temp_outer_outflow
    update_interval: 60s

Temperature efficiency

With all those temperatures available, I configured another sensor to calculate the temperature efficiency of the heat exchanger.

The formula is

$$ \eta = \frac{T_2 - T_1}{T_3 - T_1} \times 100%$ $$

$T_1$ – outdoor temperature before heat exchanger
$T_2$ – temperature of supply air after heat exchanger
$T_3$ – temperature of extracted air from rooms

The ESPHome definition:

sensor:
  - platform: template
    name: efficiency
    filters:
      - round: 2
    unit_of_measurement: '%'
    state_class: "measurement"
    accuracy_decimals: 1
    lambda: |-
      float t1 = id(temp_outer_inflow).state;
      float t2 = id(temp_inner_inflow).state;
      float t3 = id(temp_inner_outflow).state;
      float eff = (t2 - t1) / (t3 - t1);
      return eff * 100;
    update_interval: 60s

I could have done the same using a Home Assistant template sensor. However, doing it directly on the ESP allowed me to offload some work from the main instance and bring the calculation closer to the data source. The data is already available to the ESP, so it can expose the pre-calculated value as part of its API.

Deployment

The initial deployment was a bit different from a traditional software development workflow. I had to visit my attic, gently place all the sensors inside the air pipes according to the plan, and power up the Wemos D1 board.

Once everything was in place, I switched over to Home Assistant.

Home Assistant dashboard

Since I already had ESPHome integrated with Home Assistant, the new device was automatically discovered and all exposed entities were registered. I tweaked the names and prepared a dashboard to have a quick overview of my ventilation unit.

The image shows low efficiency and temperatures, that don’t look great in terms of heat exchange but no worries – that’s because the unit was operating in bypass mode at the time. I’ll provide a better image when it’s cooler.

I’m not going to show you step-by-step configuration here, but if you think it would be useful for you, just drop me a message. I’ll be happy to share the details.

Final thoughts

This setup is working flawlessly for half a year. In the next iteration, I’m going to give the ESP a dedicated case and replace the Wago connector. But once it’s working, the urge to improve it is lower than I expected.

I hope you found this write-up useful. Thanks for reading!

Test your plans by reducing time

Thu, 20 Jan 2022 00:00:00 +0000

I couldn’t find any better phrase that describes the approach I’ve learned in the last year. It’s a simple piece of advice: Test your plans by reducing time.

It’s easy to fall into the trap of thinking that the amount of our free time is stable and we’ll also be able to spend the same number of hours on things in the future.

Sooner or later, you’ll need to put most of your effort and attention in a specific direction. Eventually, you’ll push every insignificant activity in the background, and you may struggle to find even minutes to deal with the rest of your stuff. Starting of studies, new relationship, new job, starting a family – they are only examples of events that can change your priority. If you want to stick to your routines, you need to change the approach to performing tasks.

The assumption behind the main idea leads to another rule, which I named Plan for a fraction of your time and it goes as follows:

When you plan to do something, do it in a way to derive value from this activity, even if you’ll spend only 10%¹ of the time you initially planned.

Take a look at the following examples:

Do you plan to learn a new language for 30 minutes each day? What if you have only 5?
Do you want to read a book for one hour? Imagine you have only 10 minutes.
Do you have an article idea and need a few hours? Imagine you have a few 15 minutes-long blocks of uninterrupted time.
Do you do a weekly review for 30 minutes? Is it possible to do it in 5 minutes?
Do you want to journal for 15 minutes? Try to sum up everything in 2 minutes.
You have a side project – imagine you work on it only for 15 minutes per day.

Do these activities still make sense for you if you spend less time on them? If not, try to optimize them and make them worthwhile even with limited time. It helps you focus on the essential part of every task.

It doesn’t mean that you should do everything in a hurry. Instead, you should prepare yourself to do your stuff even if you have not enough time.

If there is no chance to finish a one-off task in a short time, think about producing intermediary results that are still valuable. It would help if you could track where you are and what to do next. Sometimes It’s hard to find an uninterrupted one-hour-long block, but it’s possible to find a few ten minutes-long intervals during the day. It’s not a deep-work-like experience, but the environment isn’t always perfect, and it’s your job to find a way to do your best even if conditions are unfavorable.

You may think this is a problem with an organization. Could be. You can liberate some time by abandoning some self-imposed duties. However, if you find that your routines work well for you, it’s better to keep them small rather than drop them altogether.

This article is what I’d like to tell my younger self. Now it acts as a reminder for me. Maybe also for you?

Specific value does not matter, but it should be lower than expected. ↩︎

Avoid using global git identity

Mon, 29 Nov 2021 00:00:00 +0000

When I clone a new project, I put the proper email address (e.g., a company email) as an author email in the local config stored in the repository. It worked fine until I started working for a client whose project is spread across many repositories that I cloned when needed.

I can’t count how many times I forgot to put the correct email into the local config of the newly cloned repository before committing anything into it – just because I was in a rush. In effect, my personal email (which occupies my global configuration) has been leaking many times into commits, sentenced to last forever in the git history. I’ve probably done it more times than I even realize.

To avoid making these mistakes, don’t use a global git configuration for identity.

Remove global configuration

To remove the global identity, use the following commands. If we use the same name everywhere, it makes sense to keep it set globally.

git config --global --unset user.email
git config --global --unset user.name

But this is not enough. Git will try to guess the name and email address based on the user profile and hostname. We need to instruct git to use only configuration files.

Use provided configuration only

Since version 2.8, git has a dedicated setting called user.useConfigOnly to control if it should guess the identity or use the configuration only. According to documentation:

Instruct Git to avoid trying to guess defaults for user.email and user.name, and instead retrieve the values only from the configuration. For example, if you have multiple email addresses and would like to use a different one for each repository, then with this configuration option set to true in the global config along with a name, git will prompt you to set up an email before making new commits in a newly cloned repository. Defaults to false.

The funny thing is that even documentation points out this as a potential solution to my problem.

To change this option, we can use the following command

git config --global user.useConfigOnly true

Now, if we try to commit anything to the repository that has no configured author’s name and email, git shows us the message about missing configuration.

Author identity unknown

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: no email was given and auto-detection is disabled

Now it’s hard to commit anything under a mistaken identity, but we need to configure each repository individually. To make it easier, we can prepare a few small scripts.

Prepare scripts to set up identity

The idea is simple. Whenever we get the message about the unknown identity, we can invoke a simple bash script to configure the proper profile.

#!/bin/bash

git config --local user.name "Szymon Krajewski"
git config --local user.email "git@yourdomain.com"

echo "Szymon Krajewski <git@yourdomain.com> applied as git user to this repository."

Let’s call it apply_personal_profile and make it executable.

chmod +x apply_personal_profile

You can create as many profiles as you need. I prepared a personal profile, a company profile, and one related to the client I currently work for.

I store my profiles in the dedicated directory ~/Utilities/git/profiles. I added this directory to $PATH global variable in my .zshrc, to make these scripts available from everywhere. Alternatively, you can symlink each script into /usr/local/bin, and that’s also OK.

# .zshrc
export PATH="$PATH:/Users/szymon/Utilities/git/profiles"

Apply author profile to repository

Whenever you clone or init a new repository, run one of the created scripts. If you forget about this, don’t worry – git won’t let you commit without configured email.

apply_peronal_profile

Other approaches

The simpler one

The script isn’t complicated, but it’s also not perfect. I’m sure it could be better – no duplications, basic checks, and so on – but I wanted to start small. Surprisingly, this is a single line that matters

git config --local user.email "git@yourdomain.com"

You can make an alias in your shell or search in command history for that one. The name is usually the same across profiles and can be configured globally.

Automatic

There are also tools that can do automatic profile switching based on the directory name, location, or existence of a particular file, e.g. gitconfig’s conditional includes or direnv, but I haven’t tested them. I took the most straightforward approach – improvements may come with time if needed.

How to remove hard spaces from file using awk

Sat, 07 Aug 2021 00:00:00 +0000

Some tasks seem easy when we perform them manually, but new things come to light once we want to automate them. What can go wrong if we remove hard spaces from a text file using a script?

If it’s a one-time operation, we run the editor, use the find-and-replace functionality, and that’s enough. Otherwise, we need to find a more automatic-friendly approach.

The literal approach

I’m a fan of simple tools, so I use awk for that case. Let’s say that we want to clean up the CSV file. We likely end up with something like this:

awk -F";" '{ gsub(/ /, ""); }1' "filename.csv"

If it works, we can end our solution here, but there is still one issue. How can we be sure that the provided pattern is an actual hard space?

I went into trouble because I copied the space character right from the terminal window, and after minutes of failing attempts, it turns out that the terminal printed soft spaces instead of hard ones. Be aware, that some software may also change whitespaces on their own.

Even if we successfully copied and pasted the hard spaces into our script, we need to leave a comment to let our future self notice that there is a hard space, not just space.

#!/bin/bash
#
# Hey, an important fact, this is a hard space!
#                 \ /
awk -F";" '{ gsub(/ /, ""); }1' "filename.csv"

Hex to the rescue

The gsub function expects two parameters: the pattern and string to replace. However, we can provide a byte sequence instead of just characters.

First, let’s check the byte of the hard space. Just copy the hard space from the file using the app that preserves whitespaces characters into the terminal and run hexdump.

echo " " | hexdump -C

The output should be similar to this one:

00000000  c2 a0 0a                                          |...|
00000003

We got three bytes. The last one is 0a, which corresponds to line feed and indicates the end of the line. So, the hard space consists of two bytes: c2 and a0.

Let’s use them within our script. To tell the gsub function that we provide hexadecimal values, prepend the bytes with \x.

awk -F";" '{ gsub(/\xc2\xa0/, ""); }1' "filename.csv"

Great! We’ve removed hard spaces from the file. Use it also if you need to replace them with soft ones.

Resources

https://stackoverflow.com/a/27056408 – use of hexdump tool

How to decrypt PDF files on macOS

Thu, 08 Apr 2021 00:00:00 +0000

Some companies send encrypted PDF documents. If you’re a macOS user, Preview will ask you for the password each time you use quick look or open the document. Moreover, if you print that document, you’ll end up with a blank sheet of paper. At least I got it. So secure.

To print an encrypted PDF file, you have to decrypt it first.

In macOS, you may use a command-line tool called qpdf. The easiest way to install it is to use homebrew.

brew install qpdf

Then, navigate to the directory containing your encrypted PDF file and run the following command:

qpdf --decrypt --password=PASSWORD_TO_DECRYPT ENCRYPTED_FILE.pdf DECRYPTED_FILE.pdf

If you need to decrypt many PDF files which share the same password, you can use the loop. Below is an example loop in zsh shell:

for file in *.pdf; do qpdf --decrypt --password=PASSWORD_TO_DECRYPT $file decrypted_$file"; done

Missing article image on Twitter due to GetMatch in Hugo template

Sun, 14 Mar 2021 00:00:00 +0000

Recently I received notification that somebody post information about my article on Twitter (thanks!). Twitter usually renders a link to blog posts as nice-looking cards. The card to my article shown up without a featured image.

I checked my Twitter feed, and some of my other entries also suffer from missing featured images.

Schematic responses from the API when user calls /products endpoint

After checking some of my articles in Twitter’s Card Validator, It turned out that some of them have missing metadata that twitter needs to render the cover image.

I found that my theme uses Hugo’s internal template to render Twitter’s metadata. Unfortunately, nothing within the template gave me an answer at first glance. I found out that to render twitter:image metadata correctly, there must be an image with feature, cover, or thumbnail in the filename. I follow this convention, even though I didn’t know about it.

Articles that have missing cover have one in common. Since I use Page Bundles, after migrating from WordPress to Hugo, I put images in the directory called images for each article.

As you see carefuly, there is a problem with GetMatch function used in the template. Let’s look at the following code:

{{- $featured := $images.GetMatch "*feature*" -}}
{{- if not $featured }}{{ $featured = $images.GetMatch "{*cover*,*thumbnail*}" }}{{ end -}}

According to the Hugo documentation, this specific usage of GetMatch will work only for files in the root of each Page Bundle. The path images/featured.jpg doesn’t match the *feature* pattern.

So I moved featured images to the root of each post directory. Now, my old articles have started rendering proper metadata, giving Twitter enough data to display the cover image on the feed.

Line break in Markdown

Fri, 12 Mar 2021 00:00:00 +0000

There are a few different methods to enforce a line break in the same paragraph in Markdown syntax: HTML’s <br />, backslash and double space. Let’s consider each case.

HTML tag

First of all, because Markdown syntax allows using simple HTML, the <br /> tag is one option.

This is the first line<br />
And this is another line...

Markdown, however, is a format that should be easy to write and easy to read. HTML tags, don’t matter how simple they are, in my opinion, make the document a bit messy.

My rule of thumb is simple – I should be able to read the Markdown text comfortably even if it’s not parsed.

Backslash

The second option supported by some Markdown parsers (like Goldmark that Hugo uses) is the backslash as the last character in the line.

This is the first line\
And this is another line...

Much better than the line break tag in HTML. It’s a good choice for people who write single paragraphs across many lines and need a visual indicator of the line break.

Two spaces

Fortunately, there is also an option for me. Two spaces at the end of the line also produce a line break.

This is the first line{space}{space}
And this is another line...

I don’t use to split paragraphs across multiple lines. That’s why I don’t need any particular indicator that tells me there is a line break. The actual line break is sufficient for me by itself.

Resources

Restore files from encrypted iPhone backup

Wed, 10 Mar 2021 00:00:00 +0000

I do not use iCloud backup for my iPhone. Instead, I do local encrypted backup, more or less regularly. I wanted to browse and find something from one of these backup.

Did you know, that there is no native way on macOS to browse and restore something from your encrypted iPhone backup? I didn’t.

If you have encypted backup, you’ve a problem. The Internet is full of software that solve this issue. However, I haven’t want to deal with commercial, third party software.

Fortunately, a few years ago, smart guys (Jean-Baptiste Bédrune and Jean Sigwald) found a way to access the data. User andrewdotn on StackOverflow described the situation and how to deal with encrypted backups – in theory and in practice. Ultimately, I found that GitHub’s user jsharkey13 shared a ready-to-go solution as a python library, called iphone_backup_decrypt.

I had opportunity to test the process of restoring the files and can confirm, that everything worked great!

Some context and environent:

encrypted local backup of iOS 14.4
macOS Catalina 10.15.7
Python 3.9.1 + libraries required by iphone_backup_decrypt (installed by pip3)

I utilized the example code from the repository.

If you need to withdraw data from encrypted iPhone backup, it’s the one of potential solutions.

Using local DNS in dhcpcd causes to ignore others DNS

Sat, 06 Mar 2021 00:00:00 +0000

I struggled with the strange behavior of dhcpcd on my Raspberry Pi powered by Raspbian. The RPi does many things on my local network, including ad-blocking thanks to the pihole service.

I run pihole¹ as a docker container, and it does its job perfectly well. Thanks to the port binding, I could bind port 53 (DNS) directly to the host and use its address as a DNS server.

Network configuration

interface wlan0
static ip_address=192.168.1.102/24
static routers=192.168.1.1
static domain_name_servers=127.0.0.1 1.1.1.1 8.8.8.8

interface eth0
static ip_address=192.168.1.2/24
static routers=192.168.1.1
static domain_name_servers=127.0.0.1 1.1.1.1 8.8.8.8

For both interfaces (wlan0 and eth0), I configured static IP addresses and DNS, using my local pihole service as a primary DNS. I didn’t want to rely only on my local instance because in case of any error or even docker image update, I’d end up without DNS. Unfortunately, despite the configuration, the system saw only one DNS – 127.0.0.1.

Here’s what I found in resolv.conf:

# Generated by resolvconf
nameserver 127.0.0.1

The same happened even if I reorder DNS in the configuration. Below change generated the same resolv.conf.

# ...
static domain_name_servers=1.1.1.1 127.0.0.1 8.8.8.8
# ...

I removed the 127.0.0.1 entirely from the configuration to test if it made any difference. After reboot, resolv.conf presented a proper set of name servers.

# Generated by resolvconf
nameserver 1.1.1.1
nameserver 8.8.8.8

Let’s form the problem: using localhost (127.0.0.1) address in the domain_name_servers option in dhcpcd causes the rest of DNS to be ignored.

The resolv.conf.head file as a solution

After some research and reading docs, I stumbled upon a resolv.conf.head file, which is a way to prepend an additional name server to the resolv.conf file.

I changed the configuration of dhcpcd to use public DNS only.

# ...
static domain_name_servers=1.1.1.1 8.8.8.8
# ...

Subsequently, I created a /etc/resolv.conf.head with the localhost address as a nameserver.

# resolv.conf.head
nameserver 127.0.0.1

After reboot, it turned out that my localhost address defined in the additional header file was successfully added as the first nameserver in the resolv.conf. Ultimately, it looks as follows:

# Generated by resolvconf
nameserver 127.0.0.1
nameserver 1.1.1.1
nameserver 8.8.8.8

If you ever struggle with this crazy behavior, treat it like a potential solution. If you know the actual reason or have a better solution – please let me know.

Featured photo by Erwan Hesry on Unsplash.

pihole is a self-hosted adblocker that works as a proxy for DNS queries, blocking requests related to tracking, advertisements, and other untrustworthy resources. ↩︎

Specify timezone for tests in Node to get deterministic results

Tue, 02 Mar 2021 00:00:00 +0000

In one of my side projects in Node, I wanted to test if the object with the date is formatted correctly as a Markdown entry. The setup of the test was the easiest part. However, comparing the outcome of the formatted Date object wasn’t so convenient because of the timezone. Because I live in the Europe/Warsaw timezone, I experience an offset relative to UTC.

Consider this test scenario:

test('it should parse link to valid markdown entry with title and tags', () => {
    const link = {
        url: 'https://szymonkrajewski.pl',
        title: "Szymon Krajewski's Blog",
        tags: ['blog', 'personal'],
        createdAt: new Date('2021-01-01T15:00:00Z')
    };

    expect(toMarkdown(link)).toBe("- 2021-01-01 15:00 # https://szymonkrajewski.pl # Szymon Krajewski's Blog @blog @personal");
});

After running this test case in jest I got this result:

expect(received).toBe(expected) // Object.is equality

Expected: "- 2021-01-01 15:00 # https://szymonkrajewski.pl # Szymon Krajewski's Blog @blog @personal"
Received: "- 2021-01-01 16:00 # https://szymonkrajewski.pl # Szymon Krajewski's Blog @blog @personal"

It’s a bit silly to arrange 15:00 but expect the 16:00. However, I found that setting the timezone for tests is an excellent way to get deterministic results regardless of the local timezone. Just invoke tests with environment variable TZ=utc, e.g. TZ=utc jest.

I also provide a change to package.json file to be able to run npm test with proper timezone.

"scripts": {
    "test": "TZ=utc jest",
}

The test scenario comes from my side project bkmrx.

Synchronize two collections with different representations

Thu, 12 Nov 2020 00:00:00 +0000

We get used to reading about big things – high-level architectural solutions, design patterns in action, or big data processing. However, most of our problems are small and lie at the low-level in our code. One of them is: how to synchronize a collection of entities with another collection, which is its simplified representation?

Imagine you have a collection of entities. You receive an updated version of that collection, e.g., from the API endpoint. But what if data are transferred using DTOs? We should add new entities based on data in DTO, update existing entities, and remove elements that don’t appear in the edited collection.

In this article, I’ll show you a generalized solution to this particular problem.

Context

Before we start working with the actual solution, let’s look at the problem itself. Image an array containing structures that represent the product in the shopping cart.

$cart = [
    ["id" => 1, "name" => "Apple iPhone 6", "quantity" => 1],
    ["id" => 2, "name" => "Samsung Galaxy S6", "quantity" => 1],
    ["id" => 3, "name" => "USB C-HDMI Adapter", "quantity" => 1],
];

Because many things may change simultaneously, we don’t want to process each action separately. Just imagine a frontend app where you can modify the basket purely on the frontend side and send the updated version to the API¹.

So, user sends updated version of the cart:

$cart = [
    ["id" => 1, "name" => "Apple iPhone 6", "quantity" => 2],
    ["id" => 3, "name" => "USB C-HDMI Adapter", "quantity" => 1],
    ["id" => 4, "name" => "HDD 1TB", "quantity" => 1],
];

What happened here?

User adds a new product to the cart, HDD 1TB,
User removes the product Samsung Galaxy S6 from the cart,
User changes the quantity of product Apple iPhone 6 from 1 to 2.

Now, you may ask why we ever consider it as a problem? After all, we have the updated version of the array right away. So, yes and no. Yes – because it’s an updated version of the previous collection. No – because it’s only a simplified version of the data model that lives on the API side.

The API may already have an entity called Cart with assigned Products with some additional data, like quantity in this case. That’s why, based on the updated $cart array, we need to update our underneath data model.

Basic solution

Forget about the generalization, and let’s develop a solution for this specific case. It may look as follows.

$cart = $this->getCart();
$newCart = $request->request->get("cart");

foreach ($cart->getProducts() as $product) {
    $updatedVersion = array_filter(
        $newCart,
        fn($newProduct) => $newProduct["id"] === $product->getId()
    );

    // remove Product from Cart model
    if (count($updatedVersion) === 0) {
        $cart->removeProduct($product);
        continue;
    }

    // update existing Product
    if (count($updatedVersion) === 1) {
        $index = array_key_first($updatedVersion);
        $product->setQuantity($updatedVersion["quantity"]);

        unset($newCart[$index]); // remove processed product
        continue;
    }
}

// add new elements to Cart model
foreach ($newCart as $newProduct) {
    $cart->addProduct(new ProductRef($newProduct["id"], $newProduct["quantity"]));
}

// flush changes into the database

We define each case (add, update, remove), so Cart is working as expected. Is anything we can do better? Indeed, implementation is a moot point, but we want to prepare a tool for handling similar cases without worrying about these lookups and loops.

Abstraction above synchronization of collections

The whole mechanism looks good, but to make it more generic, we need to replace specific operations with some abstracts:

How can we identify an updated version of the currently existing element?
How can we add a new element into the collection (including data model)?
How can we remove an element from the collection?
How can we update an element in the collection?

If we extract the behaviors mentioned above outside the mechanism, we should end up with a generic solution to synchronize collections, no matter what they contain.

Handling adding/updating/removing sounds like an excellent candidate to Policy, also known as Strategy pattern. The way how we can identify the corresponding element, we could provide as an argument during synchronization.

That’s an idea.

Synchronized collection

The recipe is simple: wrap your collection in a special object called SynchronizedCollection and implement SynchronizationPolicy. You can then use a method called sync to provide another array that collection has to synchronize with and instruction on finding a corresponding element.

In the below snippets, I omitted some parts of code for readability. If you look for the whole code, I published it as GitHub Gist.

SynchronizationPolicy.php:

interface SynchronizationPolicy
{
    /**
     * @return mixed added element
     */
    public function handleAdd($data);

    /**
     * @return mixed updated element
     */
    public function handleUpdate($origin, $data);
 
    public function handleRemove($origin);
}

SynchronizedCollection.php:

class SynchronizedCollection
{
    private array $collection;
    private SynchronizationPolicy $policy;
    
    /**
     * @param array $elements Array of elements to synchronize
     * @param callable $matcher Function that match element from collection to coresponding element
     * @return $this
     */
    public function sync(array $elements, Callable $matcher): self
    {
        $copiedCollection = $this->collection;

        foreach ($this->collection as $key => $origin) {
            // find updated version of origin in provided array
            $updatedVersion = array_filter($elements, fn($element) => $matcher($element, $origin));

            // if origin is not found, then handle removal
            if (count($updatedVersion) === 0) {
                $this->policy->handleRemove($origin);

                unset($copiedCollection[$key]);
                continue;
            }

            // if origin is found then handle update
            if (count($updatedVersion) === 1) {
                $index = array_key_first($updatedVersion);

                $copiedCollection[$key] = $this->policy->handleUpdate($origin, reset($updatedVersion));

                unset($elements[$index]);
                continue;
            }

            // if origin is matched against more than one element, then throw exception
            throw new AmbiguousElementException("Provided array contains ambiguous element.");
        }

        array_walk($elements, function ($data) use (&$copiedCollection) 
            $copiedCollection[] = $this->policy->handleAdd($data);
        });

        return new static(array_values($copiedCollection), $this->policy);
    }
}

SynchronizedCollection encapsulates the whole mechanism of synchronization. The policy is responsible for providing a new or updated version of entities. Under the hood, sync composes a new array and wrap it in another SynchronizedCollection. To retrieve the raw array from it, you can use e.g., toArray method.

As you can see, it doesn’t follow the standard collection interface. However, it shouldn’t be a problem to implement missing methods or to use this method to extend your collection.

Finding the corresponding element

Let’s look yet at how we approach finding the corresponding element.

$updatedVersion = array_filter($elements, fn($element) => $matcher($element, $origin));

This seemingly enigmatic piece of code uses function composition to let the user define a simple $matcher function instead of enforcing him to implement array_filter by himself. The matcher is a simple comparator and looks as follows:

function ($toSyncElement, $collectionElement): bool;

The first argument holds data to sync, whereas the second element is an actual element in the collection. Matcher should answer whether $toSyncElement concerns $collectionElement.

SynchronizedCollection in action

I refactor the previous code to use a SynchronizedCollection.

$cart = $this->getCart();
$newCart = $request->request->get("cart");

$policy = new class($cart) implements SynchronizationPolicy {
    private $cart;

    public function __construct($cart)
    {
        $this->cart = $cart;
    }

    public function handleAdd($data)
    {
        return $this->cart->addProduct(new ProductRef($data["id"], $data["quantity"]));
    }

    public function handleUpdate($origin, $data)
    {
        return $origin->setQuantity($data["quantity"]);
    }

    public function handleRemove($origin)
    {
        $this->cart->removeProduct($origin);
    }
}

$cartCollection = new SynchronizedCollection($cart, $policy);
$newCartCollection = $cartCollection->sync($newCart, fn($newProduct, $product) => $newProduct["id"] === $product->getId());

// flush changes into the database

If the definition of $policy looks too complicated for you, you can set up a dynamic policy where you can provide needed callbacks dynamically:

final class DynamicSynchronizationPolicy implements SynchronizationPolicy
{
    private $addCallback;
    private $updateCallback;
    private $removeCallback;

    public function __construct(Callable $addCallback, Callable $updateCallback, Callable $removeCallback)
    {
        $this->addCallback = $addCallback;
        $this->updateCallback = $updateCallback;
        $this->removeCallback = $removeCallback;
    }

    public function handleAdd($data)
    {
        return call_user_func($this->addCallback, $data);
    }

    public function handleUpdate($origin, $data)
    {
        return call_user_func($this->updateCallback, $origin, $data);
    }

    public function handleRemove($origin)
    {
        call_user_func($this->removeCallback, $origin);
    }
}

If you wonder why I marked this class as final, check my other article about how to use final keyword. Back to the topic, we can simplify our solution a bit more:

$cart = $this->getCart();
$newCart = $request->request->get("cart");

$policy = new DynamicSynchronizationPolicy(
    fn($data) => $cart->addProduct(new ProductRef($data["id"], $data["quantity"])),
    fn($origin, $data) => $origin->setQuantity($data["quantity"]),
    fn($origin) => $cart->removeProduct($origin)
};

$cartCollection = new SynchronizedCollection($cart, $policy);
$newCartCollection = $cartCollection->sync($newCart, fn($newProduct, $product) => $newProduct["id"] === $product->getId());

// flush changes into the database

Isn’t it simple?

Where we used it?

In one of our projects, we decided to use the Symfony Form component to handle incoming requests to the API and perform initial data validation. Instead of letting FormTypes filling entities directly, we utilized a DTO layer². But the actual problem was related to one form type, to be precise, CollectionType.

It’s a topic for a different article, but – let’s admit that – the Symfony Form component was never designed to work as an API handling method. This use-case is possible because we use only a backend side of the forms. Still, they are some visible differences between handling POST forms from browsers and API calls. A good example is CollectionType.

The CollectionType is index-based. It means that the element from the request matches the corresponding element in the same position in the collection. If you remove an element from the array and send it to the API, you unintentionally place other objects in incorrect positions. Instead of removing that specific element, the component will update every entity/DTO based on its position and remove the last one with no equivalent in the original collection. That’s why we built SynchronizedCollection – to avoid that behavior.

It’s only a brief hint of this problem. Perhaps in the future, the Form component lets us define the indexes rather than use the element’s position every time.

Summary

Surprisingly, SynchronizedCollection was settled down in our application in different places in different variations. It hides a complex problem behind a simple interface, and I think that’s the reason why it’s so useful.

The whole code is available as GitHub Gist. Feel free to make comments, improvements, and so on. I don’t think that providing a composer package is something that our community needs, but it may change in the future.

If this article was useful for you, please let me know. Btw, I haven’t found a better title for it 😶.

Featured photo by John Mark Arnold on Unsplash.

Our use case concerns collecting addresses, where the user could provide many addresses into his profile. The whole operation was contained in the update user endpoint. ↩︎
At the moment of writing this article, Symfony’s documentation has opened PR regarding usage DTOs in form handling. ↩︎

Selective serialization using serialization groups

Wed, 04 Nov 2020 00:00:00 +0000

When you ever develop an API in the Symfony framework¹, you probably heard of the serializationGroup option. It allows you to define different groups of attributes to serialize and deserialize. However, defining these groups without proper consideration may make the whole development hard, e.g., if you need to deal with different scopes of the object’s attributes in many other endpoints.

I want to show you our approach to serialization groups in one of our projects. Together with my team, we use them to control the serialization of related entities on various endpoints in our API.

Context

It’ll be more comfortable if I provide some context. Instead of talking about entities from the actual project, I’ll use a generic example: Order and Product.

Our case is a RESTish² API for a relatively small application. Some endpoints respond with entities with a relation or collection of other related entities to avoid unnecessary calls. To generate responses, we use the Serializer component. For handling requests, we rely on Symfony Form component³.

The main challenge is to return mandatory data for each scenario.

If the user calls for the Products collection, he gets a collection with basic data about each Product.
If the user calls for a specific Product, he gets very detailed data about the requested Product.

Schematic responses from the API when user calls /products endpoint

But it’s not all.

If the user calls for the Orders collection, he gets a collection with a basic data about each Order. Each order has a collection containing basic data about each Product in the order.
If the user calls for specific Order, he gets very detailed data about the requested Order. Each order has a collection containing basic data about each Product in the order.

Schematic responses from the API when user calls /orders endpoint

Product and Order are only examples. Imagine many different cases where you need to attach some extra data to a resource, but you don’t want to connect the whole object. Sometimes you don’t need to show the related items at all. Another time, in the case of other endpoints, you need to include very detailed data.

Two groups for entities are not enough

Our first solution consisted of two groups: basic and extend.

We classified each property for one of them, and they were mutually exclusive – a single field should be classified either as basic or extend. Unfortunately, this setup didn’t let us configure the properties of nested resources properly. Whenever we wanted to retrieve a full set of properties from the primary resource and some basic data from related entities, we always wind up with an extend version on each entity.

Two groups for each entity individually

To have more control over the serialization, we decided to use an entity name as a prefix for each serialization group. For Order and Product entities, it looks as follows:

order_basic
order_extend
product_basic
product_extend

Using this approach, we can serialize, e.g., the full set of properties from Order and only basic properties from its assigned Products.

Separate group for relationship

So, we split properties in each entity into basic and extend groups, but where does the collection of products belong?

Basically, it depends on the use case. However, we came up with another convention for all types of relations between entities for the sake of flexibility. Each association has assigned a dedicated group where the name consists of both owner and related entity name, e.g., order_with_products.

class Order
{
    /**
     * @Serializer\Groups({"order_basic"})
     */
    private $id;

    /**
     * @Serializer\Groups({"order_extend"})
     */
    private $createdAt;

    /**
     * @Serializer\Groups({"order_with_products"})
     */
    private $products;

    // rest method omitted
}

How we use it

Thanks to this separation of groups, we could compose responses using specific subset of properties from entities and related objects.

So, come back to the scenarios I use in the Context section. Let’s compose proper responses for each of them. I won’t describe all entities – details aren’t needed here. The @Rest/View annotation comes from the FOSRestBundle.

First, ProductController:

class ProductController extends AbstractController
{
    /**
     * @Rest/View(serializationGroups={"product_basic"})
     **/
    public function showAllProducts()
    {
        return $this->repository->findAll(); 
    }
  
    /**
     * @Rest/View(serializationGroups={"product_basic", "product_extends"})
     **/
    public function showProduct(Product $product)
    {
        return $product; 
    }
}

Next, OrderController:

class OrderController extends AbstractController
{
    /**
     * @Rest/View(serializationGroups={
     *     "order_basic", 
     *     "order_with_products", 
     *     "product_basic"
     * })
     **/
    public function showAllOrders()
    {
        return $this->repository->findAll(); 
    }

    /**
     * @Rest/View(serializationGroups={
     *     "order_basic", 
     *     "order_extend", 
     *     "order_with_products", 
     *     "product_basic"
     * })
     **/
    public function showOrder(Order $order)
    {
        return $order; 
    }
}

Summary

The serializationGroup option is powerful, not only when it comes to generating responses like this but also when dealing with different responses that depend on the user’s role or status. There’re a lot of cases when we can use it.

I don’t know how it’ll work in a bigger project with many more entities. Still, in my opinion, as long as the relations between entities are carefully designed, everything should be manageable. In our case, it does the job.

If you have more experience with it, or if you follow another approach, let me know. I’m curious how people use it in their projects.

Featured photo by Tamara Gak on Unsplash.

Or any other API that relies on the Serializer component. ↩︎
It’s not a pure REST API – we broke some REST assumptions for convenience. ↩︎
API request with data could be mapped as a standard form, so why not? ↩︎

Why deprecation process is important, on the example of restic

Sun, 18 Oct 2020 00:00:00 +0000

I use restic as my backup solution, and I haven’t had any problem so far, but recently one of my health checks suddenly stopped working, telling me that something is wrong with my backup. On the other side, restic has done its job, and everything works fine, so what was the problem?

I run the whole backup using following command:

/usr/local/bin/restic \
  --verbose \
  --files-from "$MACHINE_NAME/restic_backup/include.txt" \
  backup \
  && curl --retry 3 $RESTIC_BACKUP_HEALTCHECK_ENDPOINT

I realized that it started to occur after the last brew upgrade, and I noticed that brew also updated the restic binary to the version 0.10.0. I suspected that something changed in the restic exit code. I went through the changelog, and I saw this change:

Chg #2546: Return exit code 3 when failing to backup all source data

Oh, dear.

Yes, I back up a lot of directories, and some of them in the hierarchy are not accessible without proper permission, e.g., if you want to back up the entire APFS volume, you probably get errors as below:

error: Open: open /Volumes/Photos/.Spotlight-V100: operation not permitted
error: NodeFromFileInfo: Listxattr: xattr.list /Volumes/Photos/.TemporaryItems : permission denied
error: NodeFromFileInfo: Listxattr: xattr.list /Volumes/Photos/.Trashes : permission denied

Before version 0.10 it didn’t impact the output from a command, but for now, restic distinguishes between the full and incomplete snapshot and returns a different exit code.

How to fix it?

Exclude problematic directories and files.

Lesson learned

As a User: Read the changelog before updating software you depend on.

As a Developer: Changing the already existing behavior should be treated as breaking change, and it requires some deprecation process¹². It could be simple as below:

First step is marking the current functionality as deprecated and warn the user when he uses it. Additionally, developers may provide a feature flag turned by, e.g., environment variable to enforce a new behavior.
The second step would be replacing the new functionality with the old one but leaving a feature flag to use the old behavior.
Third step will be removing the old functionality entirely.

In the case of restic, the change was small and recorded correctly in the changelog. But it revealed the main drawback of self-hosted solutions – if anything stops working, you have to change your plans and check what’s wrong, especially if your workflow depends on the process or the tool.

I still use restic because it fulfills all my needs, and it requires little maintenance. Seriously, it was the first time I had to investigate what happened since I create my backup script. You can read how to automate your backup based on restic in my other article.

Restic still does not reach the 1.0 version, so according to SemVer 2.0, anything may change at any time. The deprecation process, in this case, depends on the goodwill of developers. ↩︎
I wrote about the need of having deprecation process but in a slighly different context. If this topic sounds interesting for you, you can read it here. ↩︎

Following in Social Media

Mon, 10 Aug 2020 00:00:00 +0000

The problem with Social Media lies in its social component, where we treat everyone as a friend. Some people take it seriously, and they share almost everything with people who often don’t really care. As long as the publisher does it consciously, know the consequences, and have fun with sharing – it’s OK. However, the problem occurs on the consumer’s side because not everyone is interested in somebody else’s life details, political opinion, religion, or other stuff that friends actively share. Why people let that content appear in front of them? Because of friendship.

Don’t confuse friendship with following

Each time we make friends with someone on Facebook, follow someone on Twitter, or connect with someone on LinkedIn, we start receiving updates from that person. Same, we let other people influence what kind of information we consume without even thinking about the quality of them. These updates are another stream of information that we need to manage, but the remedy is actually quite simple. To stay healthy in the social media world, I stick to the one specific principle that sound: following does not mean friendship .

That’s it. Following does not mean friendship. The following means that you receive updates from another person you follow. Friendship is something more profound, and I don’t even try to define it. It usually exists outside of Social Media.

Let me give you an example.

You may not be my friend, but I follow you because you publish valuable content I enjoy.
You may be my friend, but I don’t follow you because the content you post doesn’t match my current interests.

If you’re my friend and I don’t follow you, it doesn’t mean that I don’t like you. On the other hand, if I barely know you (and vice versa) but I follow you, It doesn’t mean that we are friends now – I enjoy the content you share.

Be selective

The possibility to choose who’s content we want to receive is at our fingertips. However, people are reluctant to stop following their friends, because of FOMO¹, or because it’s emotionally hard. Once you realize that the following is not the same as friendship, it will be much easier to handle the alleged loss.

Being selective for the information you may receive is much more important than you think. First, people produce way more content than each of us can ever consume, so it’s worth to choose wisely what we spend time on. Secondly, everything you read, listen, watch, and every news and update from people you follow influence you. These things shape your understanding and image of the world. How to stay happy reading continuously about calamities, deaths, climate changes, and other sad things? How to stay productive if you’re stumbling upon memes, gifs, and other viral videos everyone is sharing?

In Twitter, I follow people who publish things that resonate with me and enrich my thoughts and knowledge. It’s OK to unfollow people whose content doesn’t give you value anymore. Your interest may change in time as the people you follow.

In Facebook, the medium I rarely use, I stopped observing almost everyone besides a few friends and family members. I still have contact with people that I know, but again, the following is not the same as friendship.

LinkedIn is much more problematic. It advertises itself like a business network. For me, it works like a job advertisement website with a built-in communicator. Maybe it’ll change when I stop observing people from my network I don’t even know (mainly recruiters).

You have the power to choose. Use it. Your friends will understand, and majority don’t even notice.

Featured photo by Jehyun Sung on Unsplash

Fear Of Missing Out ↩︎

Setup end-to-end encrypted cloud drive using rclone

Fri, 07 Aug 2020 00:00:00 +0000

Last time I’ve experimented with applying various encryption solutions to the public cloud storage such as Google Drive or OneDrive. They are relatively cheap online storage. However, I don’t consider them the best place for all my files, especially sensitive ones. To make them more useful to me, I was looking for an end-to-end encryption mechanism I could use with clouds in tandem, and I finally discovered rclone.

In this article, I’d like to show you my approach to use rclone as the way to have an end-to-end encrypted cloud drive.

It’s not my first endeavour with cloud storage and encryption. Some time ago, I wrote about syncing files with cryptomator vault. Cryptomator is also an excellent tool, but for the current case, I found rclone to be more useful. Maybe I’ll write about it in another article.

‼️ IMPORTANT NOTE ‼️
This article demonstrates how to set up an encrypted cloud drive to protect your data and privacy from your cloud storage provider. I do not recommend and I do not encourage to treat it as a way to hide and store illegal files.

Why even bother?

Almost every storage provider encrypts your data (Google, Dropbox, OneDrive) or gives you control over encryption (S3). Thanks to them, your data is encrypted in transit and at rest. It means, no breach on your provider’s server will reveal your files. Your account works like a key – only you and your provider have access to the unencrypted data (because the service also has access to encryption key).

On the other hand, your credentials are the only thing that lies between the public Internet and your data. Your data is safe as long as your login and password are secure. Turning on two-factor authentication improves the protection even more, but still, there is only an authentication procedure that is under someone’s else control.

When something is end-to-end encrypted, no one but you can view the data. It’s an entirely different level of protection and privacy – even the service where you store your files knows nothing about them because decryption takes place only on your devices. It comes with a cost – the fancy web interface where you can browse files, preview images and documents, or even edit them is no longer usable. The cloud is only a container for your encrypted data.

List of encrypted files on OneDrive web-interface.

So, is it worth even bother? You have to answer it by yourself. Using end-to-end encryption on your own isn’t the most convenient way of utilizing cloud storage. You lose a lot of features it provides. Unfortunately, convenience rarely goes hand-in-hand with security, and, what is more relevant today – privacy. Remember that it’s not zero-one decision – you can still use encryption for your most sensitive data (what I highly recommend) and store other files as usual.

My motivations

I used to avoid keeping my data outside devices I’m able to control. I didn’t need to store things online because most of the time, I had my computer in my backpack, so my data was almost always available. But things changed. Mostly because:

I started using two different computers,
I slowly run out of space on my private MacBook,
I began to treat my phone as my second computer, and I concluded that it would be convenient to access the same files I keep on my laptop,
besides my backup, I wanted to have another place to store my files safely,
I pay for Microsoft365, where I have 1TB of OneDrive included – it would be nice to take advantage of it.

I decided to take simple criteria:

End-to-end encryption – data must be encrypted before leaving my device,
Hard disk is not the limitation – I should be able to store gigabytes of data, even if the size of encrypted data exceeds the size of my hard drive,
OS integration – it should be usable with Finder, e.g., using FUSE,
Hassle-free – it should be a hassle-free, or at the very least, it should require as minimum maintenance-work as possible ¹,
Open-Source – it should be an open-source solution,
Multi-platform – it should work on macOS, but shouldn’t be limited to this system – even better if it’s possible to access the data on the smartphone.

I didn’t perform exhaustive tests, but I found rclone to be the most suitable solution to build the encrypted cloud drive on top of it. I don’t answer if this solution is efficient, so if you’re looking for charts and numbers, it’s not the right place.

In this article, I’ll show you how to set up everything on macOS as I did. Still, almost everything (besides installation and launchd part) is also applicable to every other Linux system.

What is rclone?

Rclone promotes itself as a rsync for cloud storage. It lets you synchronize files between your computer and cloud storage providers or even between various cloud providers using your computer as a proxy.

File synchronization isn’t the only thing that rclone does. It also has a bunch of functionalities that allow you to expose and manipulate the data. For instance, you can mount a cloud drive directly to your operating system and use it as another drive. Or you can expose the content of the cloud drive by WebDAV, FTP, HTTP, or even DLNA protocol.

Preparation

Install rclone

First, you need to install a rclone. The simplest way is to do this is using brew.

$ brew install rclone

Configure remote storage

Rclone has built-in support for plenty of cloud storage providers. As I mentioned in [My Motivations section][#My Motivations], I use OneDrive, because I have 1TB online storage attached to the Microsoft365 subscription.

To change the configuration of rclone, run the following command.

$ rclone config

Then, create a new configuration by following instructions on the screen.

In my case, I created a new configuration named onedrive. I had to authorize rclone to use my OneDrive account. After that, I could view and manipulate my data using rclone commands.

To view the list of files in your account, use the below command (replace onedrive with your configuration name).

$ rclone lsd onedrive

You can also mount this storage into your filesystem, but on this stage, it doesn’t have any encryption attached so that it will be working more or less like any other client. The main difference is that the mounted files aren’t actually on your computer, so they don’t take up your hard drive space.

Add encryption layer

Rclone provides an encryption layer by remote called crypt. I called it layer because it’s a wrapper around another remote.

Since it’s technically possible to encrypt the whole backend, it’s not recommended practice because some providers keep special files or directories in the root directory. Also, it may collide with data you already keep on the cloud – rclone will try to decrypt them, showing you a bunch of messy files. Instead, create a dedicated directory for the encrypted data.

To add the encryption layer, go to the rclone configuration once again.

$ rclone config

Then, create a new configuration, but this time choose crypt from the list of available backends. I named it onedrive-crypt to highlight that I use a previously configured onedrive backend but with an additional encryption layer.

The terminal shows available remotes. The crypt has number 10.

Rclone will ask about the remote to encrypt/decrypt. It’s time to select the place for encrypted files. I configured it to use onedrive:/encrypted, where encrypted will be a directory on onedrive remote.

After that, you have to provide additional details related to encryption, like

file name encryption mode
directory name encryption
password and salt

Let’s talk briefly about them.

File name encryption mode

Using encryption, you can be sure that only people who know the key (probably only you) can view the content of the file. But what about its name? Does it even matter?

Name is one of the metadata. Metadata is just data about data. The name may disclose what could be in the file and plausible format, however, it’s only a guess. To avoid leaking this information, you can encrypt file names.

Rclone also has the option to obfuscate the file name. Although it hides the file name at first glance, it’s easy to recover the original title, so it’s more like an impediment to all sorts of scanners than real protection.

If you really want to hide a filename, you have to use filename encryption. It has at least two consequences.

The encrypted name will be much longer than the original file name.
Some cloud storage providers let you restore deleted files or previous versions of files. By encrypting the filenames, you may find it difficult to locate the proper file.

I use the standard encryption for file names.

Directory name encryption

Similarly to file names, you can encrypt the directory names. It’s worth considering turning on encryption because directory structure can say much more about your data than just files. It’s possible to evaluate what do you keep on the cloud by checking directory names.

Unfortunately, encryption goes with cost, and the price is the longer file path. If you want to have a complex directory structure inside your encrypted remote, e.g., your entire hard drive, you may hit the maximum file path length making your data unable to sync. When I write this article, OneDrive supports max length to 255 characters (see OneDrive-specific problems). Not so much, when encryption comes into the game. Fortunately, most providers support more characters in the file path than OneDrive.

Despite the limitation, I decided to encrypt directory names.

Password and salt

Each encryption consists of the algorithm and key. Since the algorithm is usually public, the key is the factor that secures your data. Rclone uses password and optional salt as the encryption key.

I recommend generating random strings both for encryption and salt. Rclone can do it for you. You’ll only need to choose a password strength. The recommendation is 128 bits, which is fine, but I use 256 bits – just in case.

Remember, put these passwords in a safe place, e.g., in a password manager.

Test configuration

My configuration looks as follows (I stripped the secrets):

[onedrive]
type = onedrive
token = *** JSON CONTAINING ACCESS TOKEN ***
drive_id *** DRIVE_ID ***
drive_type = personal

[onedrive-crypt]
type = crypt
remote = onedrive:/encrypted
filename_encryption = standard
directory_name_encryption = true
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***

Once you finish the configuration part, you can check how your new encrypted storage works. You can mount it to the filesystem, move some files there and check how the content looks from, e.g., the web interface of your cloud storage.

Mount encrypted volume

Since it’s technically possible to mount storage system-wide, it requires a little different approach to storing configuration. In this example, I configure rclone, launchd agent, and mount path in the scope of my user profile.

Rclone offers a few different ways to browse files on your remote, but I found it convenient to browse it as another drive using a file explorer like Finder. I created a script to mount encrypted remote with some additional options.

Open new file, e.g. mount_encrypted_storage.sh and start writing your mount command. After some attempts, I’ll end up with something that satisfies me enough. You have to change some details like the volume name and the mount directory.

#!/bin/sh

/usr/local/bin/rclone mount \
    --rc \
    --transfers 8 \
    --volname TwoDrive \
    --allow-other \
    --log-level INFO \
    --buffer-size 32M \
    --vfs-cache-mode writes \
    --vfs-cache-max-size 16G \
    onedrive-crypt:/ /Users/szymon/.mount/onedrive_crypt

Make sure you have a directory where you can mount the encrypted remote.

$ mkdir -p ~/.mount/onedrive_crypt

Make this file executable and test if it’s working correctly.

$ sudo chmod +x mount-encrypted-storage.sh
$ ./mount-encrypted-storage

If there is no error in your terminal, you should be able to reach the new volume using Finder. It should appear on your computer, or you can enter it through the mounted directory.

Depending on --log-level option, rclone prints various information into stderr. Instead of using the --log-file parameter in the command, I’ll tell launchd to redirect all information pushed on this stream into the file.

Mount encrypted volume on startup

If something should happen automatically, the first thought is usually cron, whereas we have a set of more suitable tools such as systemd for Linux and launchd for macOS.

Since I use Mac, I’ll show you how I set it up using a launchd job.

I created a new plist file called pl.skrajewski.rclone.mount.onedrive_crypt.plist. This name follows the reverse domain name notation, and it can be used to categorize and sort jobs or other stuff. I mentioned about it in the article about configuring restic backup for macOS.

The definition looks as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Program</key>
  <string>/Users/szymon/scripts/mount_encrypted_storage.sh</string>

  <key>Label</key>
  <string>pl.skrajewski.rclone.mount.onedrive_crypt</string>

  <key>WorkingDirectory</key>
  <string>/Users/szymon</string>

  <key>RunAtLoad</key>
  <true/>

  <key>KeepAlive</key>
  <dict>
      <key>SuccessfulExit</key>
      <false/>
  </dict>

  <key>StandardErrorPath</key>
  <string>/Users/szymon/Library/Logs/local.rclone.mount.onedrive_crypt.log</string>
</dict>
</plist>

Since my rclone configuration exists only in the context of my user, I want to tell the launchd to treat it as a local agent. Daemons, in contrast to agents, are global and could be executed on behalf of either root or specified user.

To register definition as the local agent, I copied it into the ~/Library/LaunchAgents/ directory and loaded it using launchctl.

$ cp pl.skrajewski.rclone.mount.onedrive_crypt.plist ~/Library/LaunchAgents/
$ launchctl load ~/Library/LaunchAgents/pl.skrajewski.rclone.mount.onedrive_crypt.plist

Once the agent is loaded, it immediately starts the rclone process, and the new resource will be available in Finder.

Using your encrypted cloud

So, now you have the encrypted cloud drive. Thanks to FUSE, you can use it as any other storage attached to your computer. However, you should know how it works and how to use it to avoid running yourself in trouble.

Reliability of filesystem

First of all, the filesystem needs to be reliable. Mounting the encrypted drive using rclone and FUSE adds another abstraction layer to mimic the filesystem – every action in this space evokes a proper command in the rclone process. Sometimes it doesn’t work correctly, especially during dealing with large files. Finder isn’t aware that it works with remote storage, and it doesn’t have any mechanism to recover from unexpected network errors or timeouts.

For day to day usage, I’ve had no problems so far. I follow simple rules:

I use mounted volume mostly for accessing the data,
- but avoid using quick look on large files, e.g., movies,
I don’t work directly on mounted volume²,
I upload large files using the command line.

Initial synchronization

Since your drive is empty, I suppose that the very first thing will be to upload some files. You can start small from a few megabytes, but If you’re going to upload a massive set of data, e.g., your entire hard drive, I recommend using CLI.

I started by creating a simple script that runs synchronization for selected directories.

#!/bin/sh

OPTS="--rc --transfers 8 --progress"

rclone sync $HOME/Movies onedrive-crypt:/Movies $OPTS
rclone sync $HOME/Archive onedrive-crypt:/Archove $OPTS
rclone sync $HOME/Music onedrive-crypt:/Music $OPTS
rclone sync $HOME/Pictures onedrive-crypt:/Pictures $OPTS
rclone sync $HOME/Utilities onedrive-crypt:/Utilities $OPTS
rclone sync $HOME/Photos onedrive-crypt:/Photos $OPTS

It doesn’t look pretty and isn’t so optimal, but for initial upload, it’s OK. Thanks to the --rc option, you can control the running rclone process. It could be useful if you need to limit the bandwidth when synchronization is underway.

Save the file as, e.g., init-sync.sh and make it executable. I recommend running it using a utility called caffeinate, which prevents the system from idle sleeping. Before you do that, you need to unmount your encrypted drive.

$ chmod +x init-sync.sh
$ launchctl unload ~/Library/LaunchAgents/pl.skrajewski.rclone.mount.onedrive_crypt.plist
$ caffeinate -i init-sync.sh

If you need to limit the bandwidth during synchronization, use a proper rc command:

$ rclone rc core/bwlimit rate=512K # limit the bandwidth to 512KB/s
$ rclone rc core/bwlimit rate=off  # disable bandwidth limitation

After finished synchronization, you can mount your encrypted remote again.

$ launchctl load ~/Library/LaunchAgents/pl.skrajewski.rclone.mount.onedrive_crypt.plist

OneDrive-specific problems

OneDrive isn’t the best choice if you want to sync your entire hard drive regularly. Below I write three potential problems you may encounter during your attempts with this cloud storage.

File Name Too Long

OneDrive has a significant limitation when it comes to max file length, which is a limitation on the entire path to file, starting with the root of the OneDrive directory. A lot of people complain about this issue.

To check the actual limitation, you can run the below command.

$ rclone info --check-control=false --check-length=true --check-normalization=false --check-streaming=false onedrive-crypt:

After a couple of seconds, the answer will be print to stdout:

// onedrive-crypt
maxFileLength = 159

The rclone documentation also point to this issue:

Cloud storage systems have various limits on file name length and total path length which you are more likely to hit using “Standard” file name encryption. If you keep your file names to below 156 characters in length then you should be OK on all providers.

The need to empty trash manually

According to rclone documentation, OneDrive doesn’t provide a method to delete files permanently.

Any files you delete with rclone will end up in the trash. Microsoft doesn’t provide an API to permanently delete files, nor to empty the trash, so you will have to do that with one of Microsoft’s apps or via the OneDrive website.

It’s worth knowing about this limitation if you’re going to sync a lot of files, and you have a lot of file removals.

Ransomware detection mechanism

OneDrive provides a ransomare detection mechanism and notifies you if you’re a victim of an attack. However, from Microsoft’s perspective, upload a large set of encrypted files looks suspicious enough to classify it as ransomware, so you’ll end up with notification about this false-positive case from time to time. Unfortunately, OneDrive doesn’t allow turning off this functionality.

Summary

Let’s do a quick recap to check if we meet the criteria:

End-to-end encryption – it was the main goal,
Hard disk is not the limitation – data are stored online, the hard drive works as cache,
OS Integration– with limited support, but it’s possible,
Hassle-free – could be subjective, but for me, it’s manageable,
Multi-platform – rclone works on various operating systems. For mobile access, I use a WebDAV resource exposed in my private VPN.

Now you know one tool to create your encrypted cloud drive, where you can store files without worrying about the security and privacy – your data will be safe as long as your password and salt are secure. Moreover, you still have security that comes by default from your cloud storage provider.

You may ask: Isn’t it an exaggeration to encrypt everything? No, if you really need it. And the need should come from you, not from the data you want to hide. If you don’t feel comfortable storing things online, but you want to benefit from that, maybe end-to-end encryption is for you. Whatever your choice, remember about regular backups – and encrypt them.

Icons used in featured image: OneDrive logo, Padlock icon by Freepik from flaticon.com.

Having a maintenance-free solution when you’re wiring everything by yourself sounds like a wish, but it’s sometimes possible. I set a backup using restic some time ago, and it just works. But yes, better to be prepared that things like that need your attention from time to time. Remember also about Murphy’s Law – if it can stop working occasionally, it stops in the worst possible moment. But isn’t it true for all services at all? ↩︎
Working directly on remote encrypted storage is something I still test. However, I use different tools to synchronize files I often need. I’ve tried to use a remote photo library (not Photos app), but the performance wasn’t satisfying. ↩︎

Rethinking writing

Tue, 14 Jul 2020 00:00:00 +0000

This blog is my third approach to regular writing. It’s almost three years old now, so it’s a good indicator, that this attempt was the successful one. Over time, I published more than 40 articles, which is many and few at the same time, especially when I’ll consider my approach to writing.

The start wasn’t easy since I hadn’t picked any niche or the main topic. I collected some ideas for articles I could write about. I narrowed the scope to the programming field, but as time went by, I realized that there are plenty of other exciting topics, which, unfortunately, don’t fit into my blog.

Now, I’d like to set up new rules for myself to liberate my writings and thoughts, making this blog my unique place on the Internet, once again.

Writing is still not easy for me because of two kinds of factors:

My ability to write comprehensively on the chosen topic
Rules I put on myself

While the first point concerns more the practising of writing, the second one is something that could support you or limit you.

At the beginning of my journey as a writer, I decided to write about web technologies, because it’s something related to my work and hobby. I had a clear vision of how my articles should look like – both when it comes to the content and appearance. Each of my writing should tell the story, so I tried to embed them in the proper context. Moreover, they should have the right structure, an eye-catching topic, encouraging introduction, featured photo attuned to the content of the article and, of course, summary.

It works well for most of the time, but the whole writing process was tiring and time-consuming. Sometimes I wanted to write a few sentences to share something exciting, but this kind of content hasn’t suited into my blog in the way I previously assumed.

Now, when I have even less time to write, I need to rethink my approach to blogging. I want to treat writing as a form of relaxation and the way of sorting thoughts. To rediscover the fun behind it, I want to loosen my rules and start to treat this blog more like a place for me, rather than the site about a specific topic. It should be the space, where I can put things I’m interested in, and you or anyone else who can reach this site, e.g. by a search engine can take advantage of the content I publish.

Out of 43 articles I published till now, a few of them gained some popularity. Surprisingly, they aren’t that articles I’m most proud of, but the ones that respond to the problem I struggled with or these which presenting one, sometimes obvious idea and my understanding of it.

So from now, instead of enforcing myself to write, I’ll more use the momentum of things I’m interested in or these which resonate with me the most. It doesn’t mean that the quality of my texts will drop. On the contrary, adding more authenticity to my writings will help me develop and release my own unique voice. Eventually, it allows me to be just me. I’m not always right, but I’m excited about the things I do.

Featured photo by Green Chameleon on Unsplash.

How to type letters with accents and diacritics in Emacs

Thu, 18 Jun 2020 00:00:00 +0000

I’ve heard a lot of good things about Emacs and Org-mode package as the approach to write and organize plain-text notes. I recently found a doom-emacs project and I decided to give it a try. Unfortunately, in the default configuration, I was unable to type polish characters. Here are two solutions, how to fix it.

Problem

The option key in macOS is used to input non-ASCII characters, e.g., polish letters like “ąęść”. However, in emacs, this key is used as a <Meta> modifier. Thankfully, emacs is highly configurable, so it’s not a problem to change its behavior, including <Meta> key binding.

Preparation

You need to locate your configuration file. I use the doom-emacs configuration framework, so my configuration file is placed in ~/.doom.d/config.el.

After each change, you should sync your local configuration with the emacs configuration using the following command. I assume that you store your emacs configuration in the default directory. Otherwise, you have to locate the doom binary in your setup.

~/.emacs.d/bin/doom sync

But first, let’s go to your configuration. You can go to the list of your configuration files directly from doom-emacs by typing sequentially SPC f P (case is important). Then, select the config.el file and press RET.

Go to the bottom of the configuration file by pressing the g key once.

Solution 1: Use command key as <Meta> modifier

Since you need option key to behave in a standard way, you need to assign <Meta> modifier to another key. The emacs wiki proposes using command key as <Meta> modifier.

Put this snippet in your configuration file and sync it with emacs-doom.

(setq mac-option-key-is-meta nil)
(setq mac-command-key-is-meta t)
(setq mac-command-modifier 'meta)
(setq mac-option-modifier nil)

In this case, you’ll lose another modifier called <Super>. If you want to keep both <Meta> and <Super> modifiers bounded, you can use the second solution.

Solution 2: Use only left option key as <Meta> modifier

I use the right option key to write accents. It means that I can still use the left option key as the <Meta> modifier. Emacs allows you to change the behavior of different keys, so you can turn off a binding of right option key.

(setq mac-right-option-modifier 'none)

Now, you can use left option as <Meta> and right option to write accents.

Have fun with Emacs but remember to get your things done!

Draw.io real-time collaboration using Visual Studio Code and Live Share

Tue, 19 May 2020 00:00:00 +0000

You’ve probably heard about draw.io which is one of the best free, open-source diagramming software. The tool is powerful. It supports many storage backends and could be integrated into other apps. But the one crucial feature is still missing – it has lack of collaborative editing.

Remote collaboration is something we need these days. Last time, Visual Studio Code announced extension called Live Share, allowing developers to share their workspace with other people. In consequences, it opens a bunch of new possibilities!

Thanks to the VSCode, LiveShare and draw.io integration, you can bring the power of collaboration into draw.io.

Draw.io collaborative editing in VSCode

Install Visual Studio Code.
Find and install aditional extensions: Live Share and draw.io integration.
Sign in to Live Share.
Create a new workspace and share the session with your friend.
Create new .drawio file.
Enjoy!

How does it work?

Live Share in Visual Studio Code allows you to create a collaboration session. Depending on the settings, you can share your workspace in write or read-only mode. Anyone, who connects to your session will see your project in its own, customized editor.

Diagrams produced by draw.io are XML files. Thanks to draw.io integration extension, you can edit them directly in the VSCode using the visual editor. If you have .drawio files stored in your workspace, others also have access to them. Each participant who has draw.io integration will see diagram instead of the XML file.

In Live Share, every change made by collaborators is sent immediately to all participants. Because diagrams are just XML files, every change in the visual editor are reflected in the file and sent to all participants. You’ll see everything in real-time!

Limitation of collaboration in draw.io using VSCode

You will not see the cursors of other participants in the visual editor.
You will see only the performed changes. It means that you won’t see the move of block but only its final position.
You may experience issues when more than one collaborator is working on the same element.

Do you know any other limitations or problems? Please let me know!

Summary

Draw.io (or should I use diagrams.net?) is one of the best tools for creating diagrams. Real-time collaboration feature would be just the cherry on top. According to the documentation, this tool offers collaboration when you use a Google Drive storage, but I’m not sure if it’s a real-time collaboration – I haven’t tested it. Anyway, the need is real. By using a combination of VSCode, Live Share and draw.io you can reach the collaborative experience with relatively low cost.

How to convince your boss to let you write tests

Wed, 06 May 2020 00:00:00 +0000

When we talk about software development, sooner or later someone starts a discussion about tests. In our world, tests are considered as something helpful and crucial to our work. However, sometimes we need to face with people who have a different opinion about it. Moreover, it’s hard to find a proper argument, especially for a person who pays you and who wants to utilize your time effectively.

How can you convince your boss to let you write tests? Should you even ask for permission?

Convince yourself first

You already test the app manually

You can compile the application, run it and check if your changes work by clicking through the app. Your feature may not work after the first trial, so you need to fix the code and go through the same process – compiling, running and checking. You already perform manual tests.

This testing process gives you feedback about your work. If the solution works and matches the requirements, you can go ahead. Otherwise, you need to go back to the development phase.

It’s unreliable to write the code and then commit it without testing if it works. Even simple manual tests are vital to confirm that your change is working correctly.

You’re responsible for writing good-quality software

Your job is to deliver working, good-quality software. Some people say about good-enough level, but it still software that is reliable, easy to maintain and develop. When the company hires you, people expect that you’ll do your best to deliver software, not worse that it currently is.

Tests are necessary to write reliable software

I don’t know any good piece of software which was writing without tests. As long as the application is small and simple, finding bugs and problems isn’t a big deal. But the more complex app, the more time you have to spend testing it manually. Ultimately, it’s almost impossible to check every affected use-case in a reasonable amount of time, so developers test only the most crucial part of the app. It should no longer be considered reliable – nobody can prove it.

Writing tests is a part of the development process

Tests, especially unit and integration tests, support you with day-to-day development. You can write them before the code (according to TDD), but you can also create them after the production code. Anyway, they are still part of the development process. You may don’t know which strategy would be better in the specific case.

You should keep the development and testing¹ parts under the same sign – software development. Mark features as done, but only if you finished both: production code and tests.

How to convince your boss to let you write tests?

Simple answer: do not do this.

If you convinced yourself first, you should know that tests are for you. Your job is to deliver working, good-quality software. How could you do that without testing if the code you wrote works correctly? Do you provide untested solutions? I know you don’t.

What would say your boss, if you would ask him if the feature you work on should be tested? Would he be surprised? For sure, he pays you to deliver working features. He expects the outcome.

People expect that they get a good quality product. In the restaurant, the chief wouldn’t ask you if the dish you’ve ordered should be tasty or not.

Writing tests should be your decision

If it doesn’t convince you, ask yourself, who will be responsible if something goes wrong? The code is the common good, so the whole team should take responsibility for each failure and mistake. Not your boss, who doesn’t let you write tests.

Tests are for you. They should help you with your work. Saying “you shouldn’t write tests, it’s a waste of time” have roughly the same meaning as “you shouldn’t use design patterns, it’s a waste of time”.

Having tests is like having contracts for requirements to the developed feature. Having automated tests is like having a partner to check if the code meets all requirements.

Remember, you already test your software, anyway. You can also automate this process. Is it in anyone business to discourage you not to test software you produce?

Don’t ask for permission to write good-quality software. It’s your job.

Change the way you communicate

We all know that tests should be an integral part of any feature we produce, any new class we write and any other development endeavour. Tests are part of our craft, not extras to it. However, this is not consistent with the way how we use to communicate the testing part to the outside world. Have you heard this?

“Yes, this feature is ready, but I have to write some tests for it.”

“It’s done, yet I need to write unit tests.”

“Everything is working, but I need a couple of hours to write automated tests.”

Since we agreed that tests are an integral part of the development process, why we talk about them as if they were a separate part? How could something be done if a part of the whole is missing?

By talking that the feature is done without tests, you communicate that development and tests are two separate parts². Moreover, if the first part could be finished and delivered without touching the latter, it shows that tests are redundant.

It’s not about hiding the fact you’re writing tests. Your goal should be not to deceive the management but to show the actual situation honestly.

Change the way you talk about tests. Treat them as a part of the development process.

Featured photo by Juan Rumimpunu on Unsplash.

Unit tests and integration tests. ↩︎
On a higher level, when we talk about the QA process, they actually could be two separate parts. ↩︎

Why I migrated my blog from WordPress to Hugo

Thu, 16 Apr 2020 00:00:00 +0000

Before you start your blog, you have to perform a set of essential choices – topics you want to write about, domain name, and the way you’ll publish your content. I choose the self-hosted WordPress platform with a slightly customized theme that I found on the Internet.

After more than two years of blogging, I migrated all my content to Hugo. In this article, I would like to show you why I decided to move on from WordPress to Hugo – a static page generator.

Why I started with WordPress

Before I tell you why I changed WordPress to Hugo, let me explain why I choose WP as my leading platform.

When it comes to writing a blog, we have two definitions:

Write a blog – write a whole blog engine from scratch.
Write a blog – regularly publish new content to share knowledge.

If you want to start blogging and you begin with creating your own CMS, blogging platform, or something like that, you’re doing it wrong. You just create another piece of software, side-project, or something that you should maintain in time. I had tried this path, and it turned out that I wanted to test myself if I can write a blog engine rather than a blog itself.

Does it mean that creating a blog engine from scratch is bad? – No, if your goal is to create a blog engine.

When I started my current blog, I decided to focus on content. I wanted to write and publish regularly, and I knew that I don’t want to spend time maintaining a platform. I thought big, but I wanted to start small. I choose the safe option – WordPress with a free, easy to customize template. Easy to modify because it’s PHP. Easy to extend because of lots of plugins. It worked.

So the reason was simple – content is more important than platform.

Why I changed WordPress

WordPress needs PHP-powered server

My blog was simple. I hosted my articles with comments feed, and I needed some space for images and other assets. Besides that, I had a few pages with extra information, e.g., contact and about me. That’s it.

Since WordPress is written in PHP, it needs a server to operate. It’s not a high cost, but it’s still the cost I incurred. For me, it was no reason to pay money for it knowing that I can reflect the same functionality on a statically generated page an host it for free.

WordPress needs constant maintenance

WordPress is a living project. It gets frequent updates, as well as its plugins and themes. It happened that I had updated them a few times per week. It wasn’t necessary, but it’s better to keep everything up-to-date.

When it comes to the themes, I used a child theme that derivates from another template. It was a rare situation, but when I updated the parent template, something break on my page. It happened twice, but I’ve never been ready to put extra time to fix it.

WordPress is well-known for bots

WordPress is a very popular, open-source project which powers millions of websites. They serve the comments system, login form, and many other things in the same way. Bots that browse the Internet know how to scrape data, write comments, and how to attack login form with some default credentials.

Thanks to plugins like Akismet and Wordfence, which mitigated the effects of bots and potential attackers, my blog instance operated normally. However, my website is so simple that I don’t want to worry about security.

My WordPress-powered blog was slow

I didn’t take care of the speed site at the beginning of my blogging journey. If you have little content and no visitors, the speed is not a factor you should improve.

I’m interested in optimization, so I knew that plenty of things in my theme could be better. Instead of investing time to fix them, I set up Cloudflare as a proxy for my website. Thanks to it, it was caching most of the resources, increasing the overall site speed in the outcome.

The result of PageSpeed Insight for my old, WordPress powered blog.

I didn’t test any cache-related plugin in WordPress, but maybe it could solve the problem with speed?

Why I moved to static page?

I heard about static-page generators way before I started my blog. I didn’t have any experience with them, but the whole idea was promising. During years, a lot of static page generators emerged, and more blogs were conducted this way.

I tested Gatsby for a while to check what it offers. However, I dropped out of it and picked the Hugo instead. I read a lot about how fast and simple it is. Unlike Gatsby, Hugo doesn’t provide any plugin system, so everything depends on the used theme. For some people, this is a huge limitation. For me, it was a nod to simplicity I like.

Ultimately, I migrated to Hugo because it fits perfectly in my workflow, whereas publishing with WordPress just limits me.

Code is a single source of truth

Hugo generates the page from files, so everything is reflected in the code base and versioned in a git repository. There is no external database providing additional configuration and admin panel, where you can change some settings. Code is a single source of truth.

If I want to change something in layout, I don’t have to look for a proper hook in WordPress lifecycle – I just go to the file and modify it. If I need to update CSS, I go to the stylesheet without thinking if Additional CSS in theme customization will be better for this change or not.

I’m a programmer, and working with the code is more convenient for me than jumping across the interface. It is also valid for my writings – I prefer to write using Markdown syntax, instead of formatting options in a rich-text editor.

One place for my writings

I used to store my writings in a separate repository to have a history of all changes. Because I published with WordPress, I had to convert articles to HTML and then put them to blog.

Writings are living things. Although I like to write timeless materials, sometimes I need to update them because my understanding of a given topic got better. Sometimes I just notice a language mistake I want to fix. Having the same content in two places was inconvenient, especially when I had to provide a change. I often forgot to move updates from WordPress to origin files.

Hugo supports Markdown. It works purely on text files, so all of my writings are in one place in one format. I can edit them using my favorite editor and publish them without worrying about how to move it to WordPress¹.

Why I migrated to Hugo?

The features I mentioned above are not exclusively in Hugo but almost in every other static page generator. However, Hugo met all my criteria: is fast and straightforward, has content powered by Markdown, its engine isn’t in JavaScript. The last one is related to the problem of longevity. I want to have something reliable for years. I don’t want to be surprised that I can’t generate my blog because of issues with dependencies after an update. The second reason was more prosaic – I want to try something new.

Migration

The whole migration process took me more time than I expected, but I should consider it as a normal situation – estimation is a hard thing. The migration process had three main parts: template migration, content migration, and deployment.

Template migration

Previously my blog was powered by a customized version of template called Themelia. It’s a simple theme, and I wanted to keep the same simplicity.

My blog on customized Themelia template.

I had been looking for the template for Hugo, and I found the project hello-friend. It’s a blog template with a clean design, very similar to what I needed. I decided to use it as a foundation for my custom theme. Its author, Radek Kozieł (@panr), has done an outstanding job – Thank you, Radek!

After a few modifications, the blog started looking similar to the WordPress version.

Content migration

Although I kept drafts of my posts in Markdown, most of them were not updated with the live version. To avoid publishing old content, I decided to migrate everything from WordPress.

In the beginning, I tried to do everything manually, since I didn’t publish lots of articles. I was coping and pasting the content from WordPress and manually changing HTML tags to Markdown and tweaking YAML Front Matter. After three posts (or a few hours), I thought that it had to be an easier way to do this.

I had installed WordPress to Hugo Exporter plugin, and thanks to it, I saved a lot of time. It did 90% of the work I had previously done manually. Great tool.

In exported articles, I had to change yet:

YAML Front Matter and settings related to the used Hugo theme
Hardcoded internal links to links generated by ref Hugo’s function
Display images by figure shortcode, instead of Markdown syntax (to have a caption built-in)

YAML Front Matter

My YAML Front Matter looks as follows:

---
title: Why should you return early?
date: 2017-11-20T09:00:25+00:00
slug: why-should-you-return-early
cover: images/Featured-Return-early.jpg
useRelativeCover: true
categories: 
 - Software Development
tags: 
 - Code Style
 - PHP
 - Software Design
---

The migration tool put the url parameter for each post. However, I prefer to use a slug instead. Properties cover and useRelativeCover are used by the template to display the post’s featured image.

Content organization

Group articles together

The migration of the content was the one problem, but how to organize them in Hugo is a different story. By default, Hugo reflects the structure of the content directory during page build. However, links to my articles consist of domain and slug – they are in the “root directory”.

I’d not like to keep everything in one single directory. Fortunately, Hugo allows configuring permalinks to the specific type of content.

So, i put all of my posts to /content/posts directory and in config.toml I configured a proper permalink:

[permalinks]
  posts = ":slug/"

Content ordering

Most of the IDEs keep file structure ordered alphabetically. When it comes to my writings, It’s more comfortable for me to order articles by published date.

I prefixed each article with the date to retain the order. Without it, old content was mixed with the new one, and everything looked messy. The one downside is that I have to override the slug property in YAML Front Matter for each post. Otherwise, the URL will contain a date as well.

Store assets using Page Bundles

Page Bundles in Hugo allowed me to keep together everything related to the single article. It means that I don’t have to put every image in static directory, which was default place for pictures and other non-processed stuff. Moreover, images stored as page resources could be processed (resize, crop, transformation), so it’s a way for future optimizations.

So, instead of keeping markdown files directly in the content/posts directory, I created a separate directory for each article and other related assets.

Comments

Comments were the big unknown for me since that was the only dynamic part on my website. However, I realized that most of the comments I got were spam filtered by Akismet. Besides that, I got a few valuable comments, which were a big inspiration for me – thank you.

I read a lot about the Disqus, and I saw the pros and cons, but I don’t want to bloat my site with ads, trackers, and so on². I would set up my own comments system, but this is another piece of software I would have to maintain, but you know – self-hosted services, although free, have a hidden cost.

I explored the Indieweb and found feature called Webmentions. It looks promising because it would let me collecting mentions about the post from external channels, like e.g., Twitter. Yet, this way of commenting is not obvious for everyone, and I would treat it more like an extra rather than a default manner.

So, for now, I don’t have comments. Maybe I’ll add them in the future. If you want to refer to my article, the best option to do it for know is Twitter – you can reach me out there by mentioning @skrajewski.

Hosting and deployment

Since I move to the static page, I don’t think I need the whole LAMP stack any longer. I paid for the hosting (not so much, but this investment was not paying back), but for now, it would be overkill.

I did a little research, and it turned out that lots of services offer free (but limited) hosting of static pages. I thought about the GitHub pages, but eventually, I landed on Netlify, mostly because they provide CI solution for Hugo websites. Moreover, Netlify manages its CDN, so it’s no need to use services like Cloudflare. It’s can also generate and renew SSL certificates using LetsEncrypt.

What is also important, Netlify handles redirects, so providing a custom 404 template or redirect from old article URL is not a problem.

What I forgot or didn’t realize

I prepared myself for most of the situations, but there are some I wish to know about it earlier. Not big deals, but worth considering.

E-mail hosting

On my old hosting, besides the HTTP server, I took advantage of e-mail hosting. Once I dropped out of it, I lost e-mails addresses in my domain. I haven’t used them so much. However, for me, they were a nice addition.

I’ve looked for e-mail providers, but I can’t find any affordable solution. It’s not crucial for me. Therefore I don’t want to spend too much money on things I don’t urgently need – otherwise, I probably would stay on previous my hosting. If you know worth recommended e-mail hosting providers, please let me know on Twitter.

Markdown is less Markdown

Markdown is simple. If you need to write text with headings, some tables, and images – it’s a perfect choice. However, the more custom features you want to implement in your articles, the less Markdown it becomes.

In Hugo, if you need to display something custom in the article, you need to use a shortcode. The one shortcode may display a responsive image, other an alert, or an infobox. But in any other Markdown editor, they have no representation, and they are just text. Moreover, ugly text.

{< figure src="057_window_keyboard_shortcuts_help.jpg" alt="The Keyboard Settings Window on macOS with the enabled shortcut on the list." caption="Keyboard Settings Window. Make sure you have \"Show Help menu\" action enabled and assigned to the proper shortcut.">}

If you want to keep your articles in Markdown, you’ll need to limit the number of shortcodes you use. Hugo allows to set up Markdown render hooks, so you can provide templates for images and link that Hugo³ uses during parsing native Markdown syntax.

Summary

The whole process was fun, and I learned a lot by migrating from WordPress to Hugo. It’s still a lot of things to do and ideas to implement and test, but it’s not crucial at this moment – the more valuable is content – articles that I produce, and you read.

Do I regret that I started with WordPress? No. Using WordPress as a platform is convenient, and if your goal is to write, you don’t need anymore. I didn’t worry about the technical details, images, and so on – everything was managed by the blog engine.

But WordPress allowed me to meet my needs. I saw that some things are not for me, others are fantastic and some others could be better. I decided on migration based on the experience, not the hype. WordPress is still a choice for people with no technical background.

Remember that the essential part of each blog is the content it serves. Not template, not an engine, but the author’s effort to assemble thoughts in writing.

I hope you’ll find value in this article.

Featured photo by Webaroo on Unsplash.

I heard about the JetPack plugin that brings Markdown support to WordPress. However, it’s not kind of plugin I wanted to use anyway. ↩︎
You can find more details about the Disqus privacy and performance concerns in Marco Saric’s article or Victor Zhou’s artcile, or by looking for phrase “Why should you remove Disqus”. ↩︎
Hugo uses Goldmark renderer, which supports this kind of hooks. ↩︎

How to boot system from USB using GRUB

Thu, 02 Apr 2020 00:00:00 +0000

Almost every modern computer has a dedicated menu to choose from which device the system should boot. If not, there is an option in BIOS/UEFI to change that. But what if both options failed for some reason?

One of my old computers runs Linux with GRUB as a bootloader. I use it from time to time, but one day the filesystem crashed, and even fsck couldn’t repair it. I prepared a USB stick with the new system, but I couldn’t boot from it – BIOS was restoring the booting order on every reboot. Moreover, ever since I installed GRUB, the shortcut to summon booting device selection stopped working.

I had an opportunity to learn how to boot from USB using GRUB.

How to boot from USB using GRUB

I assume that you already have a bootable USB drive with your favorite Linux distro. If not, you can use a tool called Unetbootin to create a bootable USB drive. In this case, I use a Ubuntu distro.

If you have your bootable USB drive, insert it into the computer.

Boot from proper device

When the GRUB screen appear, press C or ESC to go to the GRUB console. You should see the grub> prompt.

First, check if GRUB detects your USB drive. Type command ls to see available devices.

grub> ls
(lvm/ubuntu-vg-swap_1) (lvm/ubuntu--vg-root) (hd0) (hd0,msdos1) (hd1) (hd1,gpt2) (hd1,gpt1) (cd0)

The output of ls command in GRUB console.

GRUB recognizes my USB stick as (hd0,msdos1). I know it because I recently formatted it and created MS-DOS (FAT) partition.

Now, change the root to this device.

grub> set root=(hd0,msdos1)

You can check if you’re using a proper device. By using the command find and pressing TAB key, you can preview the list of files on it.

Next step is to locate the bootloader on USB stick to chain-load. In my case, the path to the bootloader is /efi/boot/grubx64.efi. You can use TAB to invoke auto-completion and type the path faster.

grub> chainloader /efi/boot/grubx64.efi

Screen from the GRUB console where I executed chainload command.

To invoke the new bootloader, run the command boot.

grub> boot

GRUB booted from the USB drive.

Summary

This process doesn’t look so complicated. However, it took me almost two hours before I finally figured it out. I tried different ways to fix the filesystem first, but starting from scratch was the easiest one. Fortunately, I hadn’t had essential data there, but it showed that it’s worth having a backup of your most important files. You have it, right?

Resources

The form may limit you

Fri, 20 Mar 2020 00:00:00 +0000

Sometimes I feel a surge of creativity. It’s a shame not to use this state to create something remarkable. However, once I sit down to my desk trying to write some words or code, I realize that I should do this in a particular way. Why? Because it’s how I use to do these things. And you know what? The form may limit you.

If I want to write about something, I usually don’t care about images, text decorations, even a title. The content is valuable, whereas the rest is supportive. But I took a form, that needs the title, sometimes images and one or two paragraphs of introduction. I have to select the appropriate category and tags. I analyze how the text fits the rest of the topics. Moreover, I decided to write in English, whereas it would be much easier for me to write in Polish.

I take a lot of notes using old-school plaintext. To be able to find them sometime in the future, I have to tag them and link to other entries. Sometimes it’s too much because the only thing I need to is to write something down.

I summary almost every book I read. I don’t reach for the next one until I write down conclusions from the one I currently read.

The form is also behaviour.

Sometimes I need to keep something online, but I don’t use public cloud services. Because of that, I have to put extra effort to, e.g. to keep my files in sync between my devices or to back up my most important data¹.

For developers: imagine, that client requests a small feature, probably one-off, but you need to write a few extra classes to fit it into your solid, object-oriented code.

This list may have no end. The form mentioned above – a set of rules, choices and good practices – is sometimes restrictive and kills the creative process. But we need to establish it. Otherwise, we’d drown in the ocean of inconsistent work surrounded by loosely floated pieces of information. The form gives cohesion and personal character to your work.

The form may limit you. When it happens, change the form.

Featured photo by Markus Spiske on Unsplash.

The less free time I have, the more I appreciate the cloud services without worry about maintenance. It’s still far for me to be fully convinced. ↩︎

Don't read later. Read it now.

Thu, 05 Mar 2020 00:00:00 +0000

I would like to confess a small lie that I told myself each time when I found something interesting on the Internet, but I didn’t have enough time to read it. It was Read later.

For sure, reading is my primary way to acquire new knowledge. I read a lot. I read books and blogs articles, observe people on Twitter, receive newsletters and, browse Hacker News. Each time I find something I consider that is interesting for me and I want to know more about it. Sometimes it’s a book recommendation, fresh news, tutorial or a new app announcement.

I didn’t want to miss that stuff.

Tools to the rescue?

Since I remember, I took advantage of different tools to store interesting content. Although the idea was right, I realized that something was broken in my workflow. Over the years, I collected thousands of things – articles, videos, whole blogs and websites, tools, and services I wished to got to know because I thought they would be useful in the future.

I didn’t have a solid habit of reading collected stuff. I read one article and add five new. Every time I started processing my list, I discovered that most of the items got out of date since they were here for more than half year or even more.

Even if I performed a cleanup, the story had come full circle.

I removed 600 articles since they no longer interest me.

This is no problem with tools

I thought I’d be able to solve the problem with the overwhelming amount of content only by using a proper tool. But to be honest, no tool can help you with a massive pile of articles if you don’t process the collected stuff on a regular basis. It’s no problem with tools.

I started thinking about why I store tons of links, but I don’t need to read them. It turned out that I needed to remember that something like that exists. Moreover, pushing it to Read Later heap gave me the conviction that I didn’t lose what I found, so I pushed almost everything there. Ultimately, I created a bad habit.

So in short, I put most of the stuff to the big box called “I wish to get to know more about it, but I wait for the time I’ll need it”. I didn’t know if I’d remember that I had them. I wasn’t even sure if I’d be able to find them. But Fear of Missing Out was stronger.

Don’t read everything

The crucial thing was to understand that I can’t read everything. It’s impossible. So if it’s impossible, why I even tried?

Lots of blogs, news aggregators, social media platforms and newsletters suggest you what you can read, watch or listen. They produce a continuous stream of content. In most cases, the content is neither urgent nor important.

The only reasonable way to deal with streams is to process fresh content without postponing only when you have enough time. If you’ll find something interesting, check it, take note if needed, and go on. But If you don’t have time, you lose nothing. It’s more than sure that you’ll find another exciting content next time you’ll process the stream.

Generally speaking, don’t postpone processing information from streams unless you’re sure that topic is your area of interests. The rule of thumb is: check it now or never.

Be aware of what you read and why

I’m no opponent of Read Later list. I still use it but in a more conscious way. What does it mean?

When I do research, I sometimes push some articles and materials to my Read Later list to come back to them in a relatively short time. I know that I seek this information, so it’s more likely that I’ll process the article. When somebody recommends me something, I usually push it to the Read Later list as well.

When it comes to casual Internet browsing, I have a simple strategy. I try to read stuff without postponing unless they’re related to one of my active research projects.

What if I find something interesting, but I don’t have time to read it? In most cases – nothing. I just let it go. If you have a similar problem, take into account that if you hadn’t visited the stream, you wouldn’t have seen this interesting stuff and for sure nothing wrong would happen. If you need this information, you’ll find this one or even better source.

Summary

The Read Later list should be your partner, not the enemy. Don’t feed it by random things you found on the Internet.

Of course, some people don’t care about the number of elements things on the Read Later list. 500 or 5000 – who cares? If you’re one of this person – it’s OK. Don’t solve a problem you don’t have.

I recently filtered 600 elements from my read later list. They were things that no longer interested me. Subsequently, I cleared my list at all.

Empty Pocket. First time after a few years.

Right now, I maintain a small running note with useful links and I take advantage of Safari Read Later feature to keep some information for a moment.

Next time, before you’ll add something to your Read Later list, Pocket, Instapaper, Bookmarks or another tool, ask yourself: Is it really for me?

Featured photo by Tim Mossholder on Unsplash.

Find and insert reference to note on macOS using Alfred

Wed, 12 Feb 2020 00:00:00 +0000

Create connections between notes is the essential part of the whole note-taking process, and for sure, it leads to discovering outstanding ideas in the future. However, it requires extra effort during writing: you have to find a proper note, copy the reference and paste it to the original memo. Some software has a built-in feature to handle it, but when it comes to plaintext notes, the implementation depends on the user.

In this article, I’ll show you how to create Alfred workflow that automates the process of finding and inserting a reference to other notes.

First of all, you need to have macOS and app called Alfred. I mentioned Alfred before in my article about running notes. It’s one of the most useful application on my computer.

Why it’s worth to connect notes?

Notes are rarely independent. Like neurons in our brains, they are more powerful if they contain connections to other notes. We can put the reference to different note when we feel, that the particular piece of information is related to another one. For sure, sooner or later, you’ll need a way to create connections between notes.

How to create a reference?

Notes should be long-lasting – that’s why I use plaintext to take them. The most common practice, in this case, is to use a wiki-style syntax to link from one note to another one. Most software, like nvALT, FSNotes or The Archive can parse this tag to create a link to the referenced note. Regardless of how old-school this method is, it’s resistant to time and doesn’t lock you in the specific service or note-taking software.

The reference itself is related to your naming convention. I put a unique DateTime marker as an ID to the name of my note. Thanks to it, I can create a reference using either full name or only ID – whatever is more convenient at the moment.

So to create a reference from one note to another one, I put the id of the referred memo in double-square brackets somewhere in the text of the first one. Here’s an example:

Example note with reference to another note visible in FSNotes app.

The biggest problem is to find the ID of the note I want to refer to. Without a dedicated feature, you need to change the context of your work, find the proper note, copy the ID and finally insert it into the original note. Let’s Alfred does this work.

Alfred workflow to find and insert reference to note

I created an Alfred workflow to simulate autocompletion of the note reference. You can set your directory for notes, so Alfred will limit the results. Whole workflow looks like below.

The view of whole Alfred workflow for Note Reference Search.

The workflow is triggered by a snippet. I use the snippet instead of keyword or hotkey to be able to pass configuration to File Filter and to avoid remembering the next hotkey. Moreover, during working with notes, it’s more natural to write directly in the editor, so the snippet looks like the best option here.

To activate the workflow, type following keys sequence: \[[. It’s easy to memorize because it looks like the beginning of wiki-style link preceded by a slash, which usually means command in some other software.

Alfred input window with prepared File Filter to look for notes.

The workflow looks for matched markdown files in the pre-configured directory. When you find the note you want to refer to, press Enter to replace snippet with the wiki-style reference. If you’re going to insert the full name of note instead, hold cmd and press Enter.

Alfred input window with filtered results.

Before you start, make sure you set the path to your notes directory in environment variables.

Workflow Configuration in Alfred where you can set environment variables.

Remember to set the notes_directory variable to limit the scope of searching.

Summary

I hope you’ll find value in this article. There’s no first time when I take advantage of Alfred to make my work easier. Properly configured and customized, it could be a good partner in your work.

The important takeaway is: you don’t need to use an all-in-one, huge application with tons of features that lock you in its ecosystem. If you can do something manually, you can also automate it on your own! This article responds partially to the post written by Christian on Zettelkasten.de.

The workflow is available in my GitHub’s repository. You can download it from here.

Featured photo by Volodymyr Hryshchenko on Unsplash.

Think big, start small

Wed, 22 Jan 2020 00:00:00 +0000

When I think about a new idea or my next goal in life, I usually think big. The goal has to be ambitious and attractive to me, otherwise, it’s not worth putting effort into it. However, big goals also overwhelm me, because I don’t know where I should start. So I prepare. And I think. And I prepare. And I think. And this loop goes on.

It changed. The endless loop of thinking about the best approach to achieve the goal or to do whatever you want means that you probably try to start with something big. To break it, just do the opposite – think big, but start small.

The big problem

The biggest problem with every idea is that we inaccurately think of the final effect as the result of a single action. We’re in the current position, so we need to do something and in the result, we achieve our desired goal.

Most of the time we think that desired outcome is the result of one single action, but we have problem to define that one.

The goal is usually big, so we change plans and refine things, looking for perfection when we don’t even know what it means and how it looks. Doing something sounds like a feasible activity, but it’s a delusion. In effect, instead of doing even small steps, we do nothing at all. We just think and prepare.

Starting big leads to procrastination

Thinking and preparing also refer to procrastination. Sometimes we excuse ourselves saying that we need to have a plan or schedule – to be on track. Or that we need to revalidate the idea a third time because something should look different. Or because it’s not a good time. Or because of reason. It’s all about avoiding failure.

Planning as an excuse

By having a good plan, you can mitigate the risk of failure. By improving your plan, you can reduce the risk even more. By constantly working on your plan, you can eliminate the risk of failure – If you do nothing, you don’t fail.

A good plan today is better than a perfect plan tomorrow. Don’t wait for an inspired ending to come to mind. Work your way to the ending and see what comes up.

Andy Weir

I don’t mean you shouldn’t make plans. Having a plan it’s OK as long as it helps you move forward. But it doesn’t perform the move – it’s your responsibility.

Start with something small

Being a famous blogger or Instagram influencer, building profitable project, developing recognizable open-source library, or collecting 10000 dollars for local charity – there are examples of thinking big. However, you can’t achieve any of these goals at night. Things last. What you should do is to start with a small action, then observe the results and ultimately improve it in time – continuously.

It’s much easier to think of our actions as improvements which has influence on result. Small actions are easy to define, execute and validate.

The idea should be ambitious and developmental, however, our steps towards the goal should be small enough to fulfill all of our needs at this moment and give us more opportunities for further development. If something went wrong, consequences will be smaller, so we can go back and revalidate our actions or update our plans according to results. Evolution, instead of revolution.

Consequences of starting small

Sooner or later: you’ll get your job done.

Things are not perfect and almost everywhere we can see the field for potential improvement. However, by starting small we create something that is usable. We can test the idea. We can create a habit. We can discard the idea if it’s no longer relevant for us. We can get feedback. We can change decision. Ultimately, by starting small you can reach your goal without even noticing that.

Maybe it’s more appropriate to say that you should act small?

Start small is not about laziness

Think big means that our idea should be big. This is the goal – something we would like to achieve in the future. Start small means that we shouldn’t do everything we think we would need at this moment. Just the small action that brings us closer to the goal. Because we don’t know what we should do next.

The space between current state and the goal is unknown. Do first step and look if it’s in a right direction. Then do next one. And another one. This is actually a journey and, who knows, maybe you’ll discover something even more important?

Thanks to Krzysztof Szulka and Dawid Kuśmierek for reading draft of this article.

Featured photo by Jukan Tateisi on Unsplash.

Quick access to any menu item in macOS apps

Wed, 08 Jan 2020 00:00:00 +0000

MacOS has a special shortcut I wish I have known them at the beginning of my journey with this system. The functionality is hidden in the menu in almost every app when you click on Help. I talk about this little search box.

This small feature may immensely boost your work, especially if you’re the keyboard-centric type of person. In short, it allows you to find and run almost every action in the active app. Press cmd+shift+/ and start typing.

Keyboard Settings Window. Make sure you have “Show Help menu” action enabled and assigned to the proper shortcut.

Meet the The Command Palette

When I’ve seen the Command Palette for the first time in Sublime Text editor, I consider it as the most supportive feature every app should have. Right now, most of the code editors have this feature built-in. But other apps don’t.

In apps like Gimp, Word, Excel, Photoshop where you have plenty of various actions, it’s hard to find a specific menu item, unless you spent some time with the app. It often happens to me – I know that an app has a feature I want to use, but I don’t know where it exactly is and which shortcut it has. I used to went through each menu in the app looking for the option. It was a waste of time.

Recently, I discovered that the macOS help menu has an input field that allows me to filter menu items simply by typing. You can use options directly from here. Moreover, it shows me where I can find the selected option in the menu hierarchy. Some of them have keyboard shortcuts assigned – it’s also an opportunity to get to know them. So, if you don’t know the shortcut or if the option doesn’t have the shortcut assigned, searching in Help is another fast way to invoke the desired option.

A few keystrokes are enough to find desired action. You can invoke it directly from the filtered list. Moreover, macOS shows where you can find selected menu item.

Remember, by using cmd+shift+/ you’ll get focus on Search Box in macOS help menu, where you can find any menu item available the app.

You may also use Alfred

If you use Alfred, there is a dedicated workflow called Menu Search created by this guy. Thanks to this, you can search for menu items from the active app directly in the Alfred window. It’s useful, if the app overrides the native macOS shortcut.

This Alfred workflow works similarly to the native Help menu.

I use Alfred as the replacement for Apple’s Spotlight and I found it much useful because of custom workflows, snippets, clipboard history and more. I described one of my workflow in the article about running notes.

That’s it. Thanks for reading. I hope you’ll find value in this article!

Featured photo by Mathew Schwartz on Unsplash.

My 2019 Annual Review

Tue, 31 Dec 2019 00:00:00 +0000

The end of the year is presumably a good time for reflection and review. I don’t have any strict structure for annual reviews. I did them many times before, but I haven’t found any process that I stick to. I used to set goals and check if I accomplished them. However, this year I tried something different.

Past Year Review

Although I set goals for 2019, I don’t think they’re something that drove me throughout the year. Moreover, I didn’t think too much about them. For 2020, I decided to conduct a Past Year Review by Tim Ferris.

The whole Past Year Review process fits in these 5 steps (from Tim’s blog):

Grab a notepad and create two columns: POSITIVE and NEGATIVE.
Go through your calendar from the last year, looking at every week.
For each week, jot down on the pad any people or activities or commitments that triggered peak positive or negative emotions for that month. Put them in their respective columns.
Once you’ve gone through the past year, look at your notepad list and ask, “What 20% of each column produced the most reliable or powerful peaks?”
Based on the answers, take your “positive” leaders and schedule more of them in the new year. Get them on the calendar now! Book things with friends and prepay for activities/events/commitments that you know work. It’s not real until it’s in the calendar. That’s step one. Step two is to take your “negative” leaders, put “NOT-TO-DO LIST” at the top, and put them somewhere you can see them each morning for the first few weeks of 2019 (2020). These are the people and things you know make you miserable, so don’t put them on your calendar out of obligation, guilt, FOMO, or other nonsense.

I changed this process making it more suitable for me. Instead of using notepad, I wrote down everything in a text file. Instead of working with the calendar, I went through my journal.

At the end of this step, I create two lists. Let’s call them positive and negative.

Analyse positive and negative lists

I had a lot of points on my positive list, but most of them repeated over time. They were related to other people, trips, sport, learning, creating, reading, or doing something for others.

It was hard for me to fill the negative list, so I ended up having a few specific points. In general, I pointed out things like implicit expectations, arguments, misunderstandings, excuses, as well as traffic jams, eating junk food, and checking blog stats or social media.

What is interesting, all these points are not related to income, profits or any other material benefits. They are about positive and negative emotions and feelings.

The next step I’ve done was transforming the lists to the form of do more and try to eliminate. Here they’re:

Do more

Take care of important relationships in your life, spend more time with friends and put real value in their life.
Write more articles and essays on the blog and be worried less about the form (like in this article).
Do good things and support other people.
Learn new things, read books and develop new skills.
Take care of yourself – sleep more, exercise, and check your health regularly.

Try to eliminate

Excuses.
Pretending that everything is OK instead of talking to solve problems.
Checking social media and blog statistics after a new publication.
Browsing the Internet aimlessly.
Wasting time in traffic jams.

Next steps

I should schedule some activities or events, but I don’t think I have to do it right now. I can’t plan everything I want to do at this moment. Instead, I’m going to collect some ideas related to the do more list that I can use in the next months. I want to also be more aware of my try to eliminate list, so I put them in a visible place.

What about the goals?

Before 2017, I was defining goals somehow.

For 2017 and 2018, I defined goals according to the S.M.A.R.T. rule.

In the meantime, In 2018, I defined my values.

For 2019, I defined goals based on my values and my “whys”.

In the meantime, In 2019, I defined my own mission and my vision.

For 2020, I try “Past Year Review”. At this point, I define no annual goals. Maybe I would set them when I’ve needed them.

Goals appear and disappear, become important and less important – depends on the situation. It doesn’t mean I don’t have personal goals. I have, but they are no results of annual review but rather my direction through life.

Summary

In 2019, I accomplished most of my goals. On the whole, lots of good things happened to me this year. I had many opportunities to learn, challenge and to broaden my mind. I appreciate that.

On the other hand, I know, that I’ve could do many things better. There’re always fields to improvement. The point is to see them and be aware of them.

Thank you for reading. I wish you a Happy New Year!

Featured photo by Nathan Dumlao on Unsplash.

Running notes and information capturing on macOS using Alfred

Wed, 18 Dec 2019 00:00:00 +0000

List of books. Inspiring food for thoughts. Things I’ve done today. Big ideas that can help the community I live with. That’s are examples of running notes – useful notes, that I update all the time.

This is one type of notes I use in my plaintext personal note-taking system. In this article, I’m going to show you, how to make use of them in an effective way – using tool called Alfred.

I found the name “running note” in the prof. Aleh Cherp’s article “Notes: what’s in the name?” which inspired me a lot. I’ll use that name since there is no reason to invent a different term for the same definition.

Why you may need running notes?

It’s possible, that you already have some kind of running note. If you have a list of books to read, ideas for gifts for your wife (or husband), bookmarks in your web browser, to-do list – all this information could be the note.

Sometimes I want to store a piece of information that I found or read but it’s too small to be a single note. I often do this during my work or focus time, so it shouldn’t interrupt my flow.

By keeping various lists of things that matter for you (you store them, so they matter), you may come back to the list and find something useful whenever you need it.

What is running note?

Running notes are files in your note-taking system that you update quite often. You should be able to find and edit it without breaking your workflow.

That’s it. As you can see, the idea is super simple.

Here are some examples of my running notes:

bkx – books I want to read.
fftx – quotes from movies, books, articles.
wtfx – things I wait for.
idsx – ideas.
urlx – list of my favorites sites, articles, blogs, etc.
artx – ideas for blog posts.
inbx – everything that is worth to remember but need further processing.

The names of these notes are short. I use search a lot and I don’t want to type too much. For instance, I often double-check what is the right tire pressure for in my car while I’m standing next to the compressor. I have this information in one of my running notes.

Also, the x at the end of the name means, that there is a running note.

Structure of note

It’s good to define a small structure to make further automation easier. I store each entry as a list element. Almost every markdown editor displays a good-looking list.

But sometimes it’s not enough. In some notes, I put a date before each entry to remember when I store the piece of information and how long it lives in the note. If you need more detailed information, (e.g. in the work-log file), you may use a timestamp, instead of a date.

This is how my running notes look like:

created: 2019-11-05 12:00
tags: running-note

# Ideas

- 2019-11-05 # Write about your approach to note-taking.
- 2019-11-12 # Describe the idea of running notes.
- 2019-11-12 # Describe the plaintext approach.

Don’t worry about the header. I put it in all of my notes. I’ll touch this topic in another article.

How to write to running notes?

When it comes to writing, the rule of thumb is: find the shortest path to write something in the note and avoid being disturbed. Automate as many things as possible.

If you work in the terminal, make a script that echos your input to the proper file.
Use the editor that allows you switching between files using the keyboard.
Create a snippet or abbreviation to reduce the number of keystrokes.
Use a tool like Alfred or Hammerspoon, to create a workflow that streams your input into the proper file.

Although I use the terminal a lot, I don’t have special scripts for my running notes. Instead, I do everything I need using Alfred’s workflow and text editor.

Create a workflow for Alfred

If you don’t know Alfred, it’s something like Spotlight, but with some extra features. One of my favorite Alfred’s features are Workflows. It allows you creating integrations between apps, files, scripts, and the operating system. Unfortunately, it requires Alfred’s Power-Pack which is paid.

Hammerspoon looks like a free alternative for Alfred’s Workflows, but you should have a basic programming knowledge to write your automation.

I’m a big fan of "think big, but start small" approach, so in my opinion, instead of figuring out how to create a highly-efficient, customizable, and flexible workflow, create one with some keywords and pass the argument as the input to the proper file. Simple enough.

Here’s an example workflow:

Simple Alfred workflow to store argument in the proper note.

The goal of the workflow is clear – store information as fast as possible without losing focus on what I’m doing. Take a look at the keyword inbx – it corresponds with the running note with the same name. It’s easy to remember how to store the information and where I can look for it later.

Handle more running notes

In previous example, I decided to use a variable to store the file name instead of using it directly in the “Append to File” task. Here’s why:

This workflow handles more note types by different keywords.

By using the variable you can modify the destination path, so there is no need to copy the entire workflow to support more files. You can simply add new triggers and pass the execution to further processing.

When you summon Alfred, you may type one of your keywords followed by your short note. In result, it stores information in the linked file. Alfred is accessible from any place in the system, so whether you browse the internet, chat, or write, you may call it and store whatever you want.

If you urge doing something better because you won’t bear this kind of duplication or so, or you want to implement more features because you think you’ll need them – take a break. Do you want to invest your precious time to develop something you may use? Think big, start small, improve in time.

You can preview how this workflow works:

Alfred Workflow for running notes in action. Text editor is visible only for the presentation purpose.

Summary

The process of capturing information to notes should be quick and easy. The slower the whole procedure is, the more reasons you’ll find why you shouldn’t do this. Lack of time or fear of losing focus is only a few examples.

By making the capturing process easy, you’ll see that you store more and more information – food for thoughts, gift ideas, links, moments during the workday when you need to switch the context (to let your boss realize that this way of work is inefficient). I update some of my running notes almost every single day.

Although the whole note-taking needs some of your time, it’s like an investment you do for yourself. By using appropriate tools and approaches, you may reduce the needed time and still hold benefits from taking notes. It’s worth to do it.

Featured photo by STIL on Unsplash.

Stop using specialized vocabulary

Wed, 04 Dec 2019 00:00:00 +0000

Each professional group has own specialized vocabulary. It consists of words often related to tools, technologies or processes. They make everyday communication smooth and effective. Yet, this way of communication is appropriate only for specific environments.

Sometimes IT professionals forget about their specialized vocabulary and use them in conversations with people from departments such as sales, marketing, HR, or management. Moreover, I often hear some technical words in talks with people who have nothing in common with IT. The most common view at this moment is the embarrassment drawing on people’s faces.

Would you like to talk to someone who doesn’t understand what you’re talking about?

Imagine the situation when you have a medical visit. The doctor gives you a diagnosis using complex medical vocabulary. He also adds some Latin words because he usually uses them with talks with other doctors. How would you feel? Is it your fault that you don’t understand him?

If you’re the programmer, you know – you cannot use a new word in the source code counting that the compiler will understand what you expect. You don’t do this with computers, so why do you try the same with people?

Language is a tool. If you use a tool dedicated to communication and people don’t understand you, you use it wrong. The key element of each conversation is the mutual understanding. It’s a pleasure moment when I feel, that you understand me and you feel, that I understand you. Let your conversation partner feel this way and the magic will happen.

I propose you an exercise. Try to explain hard things to people, using simple words. It could be more difficult than you expect but the outcomes could be outstanding.

Featured photo by Mimi Thian on Unsplash.

Why it's worth taking notes in plaintext

Thu, 14 Nov 2019 00:00:00 +0000

For most people, keeping notes and other stuff in text files is ridiculous – tons of services and apps could do it better. But what the better means?

Despite the availability and diversity of note-taking solutions on the market, I haven’t found any that would meet my expectations. I tried to stick to the one app but I had felt the internal resistance to settling in one specific, often a vendor-locked solution.

The breakthrough was when I ask myself – why I even want to note this information? I had started to look for my “why” and it turned out, that I don’t need a fancy app or service to storing notes but a directory called “notes”.

Why do I take notes?

Most of the people take notes – on a piece of paper, post-it notes, journals, smartphones, apps and so one. But why do we do this? I assume, that the common reason is “To remember X”. It’s a far-reaching simplification. In general, we take notes to make the information available for our future self.

If you write note that you’ll need to buy milk tomorrow, it’s a reminder for tomorrow’s self.
If you take notes from the lecture, you’ll want to learn from them later.
If you write a journal, you expect that information you log might be helpful someday.
If you find interesting information, you’ll store them hoping it’ll be useful in the future.

I hope you got the point. Each note is a piece of information addressed to somebody, including your future self.

Why plaintext?

Geared with the knowledge why I take notes, I can define my needs and check if the specific approach, app or service works for me. It turned out that the best option for me is to keep notes in plaintext.

Plaintext is easy to search

Because my notes are addressed to myself from the future, I would like to be able to find them exactly when I need them. So the searchability is the crucial factor of the whole note-taking system. Does it make sense to store notes you can’t find?

Text files are search-friendly – you may look for files containing specific words or phrases. Pictures and diagrams, although still useful, need to be additionally described, if you want to look for something they show.

Files are easy to create

New notes should be easy to create. I don’t want to run a huge program to write a few words. I have an idea – bang! – a few keystrokes and it’s stored. Have you found anything interesting? Open app using the shortcut, insert what you’ve found, and add context if needed. Get back to work. That’s it.

Plaintext is long-lasting

I usually store everything I find interesting to me. The problem is that I don’t know when I’ll need this piece of information. Will it be tomorrow? Next month? When I get a new job? When I get retired? Because of that, my notes have to be long-lasting.

Software evolves and many tools I used in the past are no longer available. The company, that provides services, may change strategy or business model e.g. by introducing payments or changing policies, making the service unattractive in the outcome. Also, sticking to the proprietary format makes you dependent on the specific tool. However, plaintext is just text.

Plaintext doesn’t depend on specific software

Data, although yours, is accessible only through the one specific piece of software provided by the company, whose vision may evolve in time. You can export them but it often means losing a specific set of features or formatting in the result. The plaintext is universal and each device and operating system support it.

Keeping data in plaintext is free. There is no subscription fee for the service or app. You can use your favorite text editor or you can buy one if you want something extra, but it’s optional.

Plaintext gives you more control and more possibilities

Because data is under your control, you are not obliged to store each piece of information on the vendor’s infrastructure. You may keep everything locally or use more privacy-focused solutions that use end-to-end encryption. You can still store your notes in the cloud (self-hosted solutions have lots of hidden costs), but it’s only your choice.

By storing data in files, you’re not limited by the one app. You can use any other tools to develop the functionality you need. For instance, if you would like you to track the changes in your notes, you can use git with launchd (or cron), to automate this process.

I talked only about encryption and versioning, but there are more possibilities. I mentioned what I use because for now, they’re only things I need.

Should it be just plaintext?

The plaintext is more like philosophy. It’s a way for organizing information and knowledge without worrying about privacy, price and compatibility.

To make this system useful, you need to dive into it. Notes will be eventually a dump of your brain that you can trust. But to make this happen, you have to make your notes a part of your life. It’s much easier if you have a private space for that, without distracting, collaborative features.

But sometimes plaintext is not enough. And that’s fine! Of course, it’s much easier to store some kind of information as images or spreadsheets instead of trying to fit each piece of information into the text form. But it’s important to remember that you have that piece of information somewhere.

Summary

A lot of things still need further explication. How to start? How to organize files? One directory or many? How to name the note? How to organize them efficiently? I’ll touch these topics in the next articles, but for now, I’m going to leave two useful resources:

If you don’t take notes but you want to start, I recommend you to think big, but start small. Start with creating a daily journal or work log. Make yourself comfortable with taking notes. Don’t try to develop your perfect note-taking system, because, on this step, you don’t even know what you need. Make yourself comfortable with taking notes first, and go to the next level if you’re ready. And don’t forget about your why – your future self may thank you someday.

Featured photo by Glenn Carstens-Peters on Unsplash.

The selfie stick story

Wed, 30 Oct 2019 00:00:00 +0000

One of the problems during traveling in pair is to take photos of us together. Since we have enough photos “from hand” in our albums, I decided to buy a selfie stick.

Instead of a standard stick with a jack wire, I bought a Bluetooth version of it. I was trying to be modern, but I forgot that I have to charge it from time to time and connect it to my phone each time I want to use it.

I regret that I had bought this selfie stick.

In the meantime, I changed my phone to one which has no headphone jack. Last holidays I noticed that I’m still able to use my stick because it has a Bluetooth connection, not a headphone jack wire.

I appreciated that I had bought this selfie stick.

Featured photo by Adam Birkett on Unsplash.

Automate your macOS backup to Backblaze B2 using Restic and launchd

Thu, 17 Oct 2019 00:00:00 +0000

In this article, I would like to show you how to prepare the script that will back up your important files to the cloud storage using a tool called restic. Even though the article is specific for macOS and Backblaze B2, you can use some techniques for creating a similar script for Linux systems.

Disclaimer

I’m not going to give you a ready-to-use script or working solution. I know nothing about your needs, your files, limitations of your network, budget, and other things that could have an impact on the backup workflow. You should always adjust the tools and the workflow to your needs.

Instead, I’ll show you my process of building the backup script that works for me. I use a word process, because it’s not a single task. Trying to build something perfect first off will lead to a product with unnecessary features. Start with the minimal script and change when you notice that you need something more.

Requirements

In this article, I’m going to use some tools and services and I assume, that you have basic knowledge of them. If not, don’t worry. I’ll describe how to use them in the context of creating the backup script. If you want to know more, please check the documentation of these tools.

Restic – a command-line tool for creating backups and restoring files.
Backblaze B2 – Cloud storage (you can use a different provider).
launchd – macOS’s system service management (you can use cron).
macOS keychain – secure way to store secrets.
Bash – I’m going to write the script in bash.

Preparing environment

Make your file structure backup-friendly

Before you start, I recommend you to clean up the disk a little bit by separating stuff you care from the clutter you don’t need. Are you need to backup your entire disk? Do you want to pay for storing all these downloaded stuff?

Keep files you want to backup and non-important files separate.

Maintaining a good and solid file structure may help you also during restoring files from snapshots. If you have a dedicated place on the disk for stuff you care, you’ll never have a problem with finding the file you lost or need.

Thanks to restic and your perseverance to organize files, you’ll add extra dimension of your disk – time. It means you are not limited to browse files only within directories, but also in different moments in time.

Install restic

First, you need to install restic. In macOS, the easiest way to do is to using homebrew.

> brew install restic

After installation, you can run the program to see available commands.

> restic

restic is a backup program which allows saving multiple revisions of files and
directories in an encrypted repository stored on different backends.

Usage:
  restic [command]

Available Commands:
  backup        Create a new backup of files and/or directories
  cache         Operate on local cache directories
  cat           Print internal objects to stdout
  ...

Initialize restic repository in Backblaze B2

This step depends on where you want to store your repository. I use the Backblaze B2 service, but you may use different storage. Here is a full list of supported storage providers.

Create Backblaze’s B2 Bucket

Create a new private bucket called e.g. restic-backup-2019. Then, In the App Keys section, add a new application key called e.g. macos-restic-backup with read and write access to this newly created bucket.

Use keys to initialize the repository

Restic reads Backblaze credentials from environment variables. Let’s export them:

> export B2_ACCOUNT_ID=<keyID visible on app keys page>
> export B2_ACCOUNT_KEY=<applicationKey displayed after app key generation>

Now, you can initialize the new repository using restic init command.

> restic init -r b2:restic-backup-2019:backup
enter password for new backend:
enter password again:

The -r flag stands from repository. The prefix (b2) points to the provider, whereas suffix is a path to the repository within the bucket (/backup).

Restic will ask you to enter a password for the new repository. Set the strong password and store it in a safe place, e.g. in a password manager such as KeePass or Keychain.

Create your first backup

Once the repository is initialized, you may create your first backup. To see how restic works, let’s do it by hand. This command creates a new snapshot of ~/Documents directory in the repository.

> /usr/local/bin/restic backup -r b2:restic-backup-2019:backup --verbose ~/Documents

First, restic will ask you for the password to the repository. If you want to automate this process, you have to provide the password differently.

Restic has two options for retrieving the password to the repository.

Using file – you can store the password in the file.
Using command – you can specify a shell command to get the password.

Retrieving a password using command sounds better for me because I can store the password securely using macOS Keychain. Then, I can obtain it using security, which is a command-line interface for the Keychain. Let’s do this way.

Store secrets in macOS keychain

You can add new entries to the keychain using the graphical interface or by the security tool. In the terminal, add a new password using the security add-generic-password command.

> security add-generic-password -s backup-restic-repository -a restic_backup -w
password data for new item:
retype password for new item:

To retrieve the password from the keychain, use the security find-generic-password command:

> security find-generic-password -s backup-restic-password-repository -w
the_secret_password_to_secure_my_backups

It’s reasonable to store other secrets in Keychain as well, so let’s add them.

> security add-generic-password -s backup-restic-repository -a restic_backup -w
> security add-generic-password -s backup-restic-b2-accound-id -a restic_backup -w
> security add-generic-password -s backup-restic-b2-account-key -a restic_backup -w

Prepare script to automate backup

Use restic in script

Let’s use secrets from Keychain to run the first backup using the dedicated script.

#!/bin/bash

export B2_ACCOUNT_ID=$(security find-generic-password -s backup-restic-b2-accound-id -w)
export B2_ACCOUNT_KEY=$(security find-generic-password -s backup-restic-b2-account-key -w)
export RESTIC_REPOSITORY=$(security find-generic-password -s backup-restic-repository -w)
export RESTIC_PASSWORD_COMMAND='security find-generic-password -s backup-restic-password-repository -w'

/usr/local/bin/restic backup --verbose -o b2.connections=20 ~/Documents

As you can see, I populate environment variables that restic uses by secrets obtained from Keychain. Only the RESTIC_PASSWORD_COMMAND variable is a string with the command because restic will run it when the password is needed.

The extra -o b2.connections=20 flag is a specific option for the Backblaze B2 backend. You can omit this if you use a different storage provider.

Save it as a backup.sh using your favorite text editor and make it executable.

> chmod +x backup.sh
> ./backup.sh

If you used a GUI to add secrets to the keychain, you should allow security obtaining passwords. Use Always Allow to avoid this question in the future.

Specify the list of directories to backing up

You can type directories or even single files directly in the restic backup command, however, if you have lots of them, the command will be hard to read and manage. Unless you don’t back up the whole home directory, it’s useful to prepare a file containing directories to backing up.

Create a new file called backup.txt with a list of directories and files you want to include in backups. Here’s an example file. All the paths are relative to the user’s home directory.

> cat ~/backup.txt
Desktop/
Documents/
Movies/
Music/
Photos/
.config/
.ssh/

Now, change the backup command to use this file. I assume, that this file is located in the home directory.

/usr/local/bin/restic backup --verbose -o b2.connections=20 --files-from ~/backup.txt

Avoid running backup twice at the same time

Because you’re preparing the automatic script, it’s worth to add some checks to prevent malfunctions, e.g. invoking script when another backup is in progress.

You can use a simple approach with the PID file in the home directory. The script could check if there is any backup process running. If so, there is no need to perform a backup, so you can finish the execution. Otherwise, the script creates a PID file and removes it once the backup is done.

PID_FILE=~/.restic_backup.pid

if [ -f "$PID_FILE" ]; then
    echo $(date +"%Y-%m-%d %T") "File $PID_FILE exist. Probably backup is already in progress."
    exit 1
fi;

echo $$ > $PID_FILE
# restic execution
rm $PID_FILE

However, it was not enough. Sometimes, because of logouts, computer restarts, or other errors, the script won’t reach the rm command and the next scheduled backup had no chance to start. You can change the approach a bit and add an extra condition to check if this specific process exists. Let’s add a date and time to the output to provide more context.

PID_FILE=~/.restic_backup.pid

if [ -f "$PID_FILE" ]; then
  if ps -p $(cat $PID_FILE) > /dev/null; then
    echo $(date +"%Y-%m-%d %T") "File $PID_FILE exist. Probably backup is already in progress."
    exit 1
  else
    echo $(date +"%Y-%m-%d %T") "File $PID_FILE exist but process " $(cat $PID_FILE) " not found. Removing PID file."
    rm $PID_FILE
  fi
fi

echo $$ > $PID_FILE
# restic execution
rm $PID_FILE

Create a backup from trusted networks

For some reason, you may don’t want to perform backups from public or untrusted networks or these created as personal mobile hot-spot, because of the limited bandwidth. In that case, it’s worth to check the active connection and allow executing script only if a computer is connected to the allowed network.

For backup purposes, you may use your home and work Wi-Fi network. To check the current Wi-Fi on macOS, you can use the networksetup -getairportnetwork en0 command.

if [[ $(networksetup -getairportnetwork en0 | grep -E "Home-Network|Work-Network") == "" ]]; then
  echo $(date +"%Y-%m-%d %T") "Unsupported network."
  exit 3
fi

By using grep with regex option you can check if the output contains one of the allowed Wi-Fi names. If not, the script exits with status code 3.

I use different status codes to distinguish why the script ends – it may be useful in future automation.

Create backup only if the computer is plugged in to power source

Backup may be a power-demanding operation, especially if restic needs to perform a full scan of files and send lots of them to the cloud. To avoid battery draining, the script should do backups only if the computer is connected to the stable power source.

You can check if your mac uses power from the battery using pmset -g ps.

if [[ $(pmset -g ps | head -1) =~ "Battery" ]]; then
  echo $(date +"%Y-%m-%d %T") "Computer is not connected to the power source."
  exit 4
fi

Schedule automatic backup

The script is almost done so you can schedule the backup by defining a launchd job. You can also use cron, but launchd is preferred way to schedule actions in macOS.

Let’s take a look at how we can approach this.

The problem with launchd

Launchd has an interesting property called StartInteval that lets you create an agent that runs a task every n seconds. Moreover, it counts time also when the computer is asleep.

<key>StartInterval</key>
<integer>300</integer>

Nevertheless, the script has some extra conditions that may break the execution earlier – this is a variation of the return early approach. Despite that, launchd get the information that script was executed and it will schedule the next execution and you end up with no backup.

I’ve experienced also another issue related to the StartInterval property, especially if the value is greater than a few hours. The job will be executed after an unpredictable time, sometimes much later than the interval value. For short intervals, like 5 minutes, everything works like a charm.

Backing up files every 5 minutes is too aggressive and surely not needed, because it leads to many snapshots of data with small differences.

I’m going to show you how to add this kind of logic into the backup script.

Add time condition to your script

First, you need to have an additional place to store the timestamp. Let’s keep it simple and use the file as we do with the PID file.

TIMESTAMP_FILE=~/.restic_backup_timestamp

This file will contain the timestamp of date in the future, but for now, it doesn’t even exist. Don’t care about it. In the script, you can add a condition to check if the timestamp file exists and if so, you may check if the current timestamp is less than the value from the file. If it’s true, then the minimum threshold hasn’t been reached and there is no need to back up – it’s too early. Otherwise, we can go further.

if [ -f "$TIMESTAMP_FILE" ]; then
  time_run=$(cat "$TIMESTAMP_FILE")
  current_time=$(date +"%s")

  if [ "$current_time" -lt "$time_run" ]; then
    exit 2
  fi
fi

You have to also set this threshold after the backup command.

echo $(date -v +6H +"%s") > $TIMESTAMP_FILE

Above statement adds 6 hours to the current time and stores the timestamp to the $TIMESTAMP_FILE. If you want to change the frequency of backup, you can replace 6H by a different value in the date command.

Create a launchd job

Now, you have to create a new property list file. In my case, the name is pl.skrajewski.restic_backup.plist, with is the same name as the label for the job. This name follows the reverse domain name notation, which is a simple way to categorize and sort not only jobs, but also packages, components and other stuff.

My plist file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN
http://www.apple.com/DTDs/PropertyList-1.0.dtd>
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>pl.skrajewski.restic_backup</string>
    
    <key>Program</key>
    <string>/Users/szymon/scripts/backup.sh</string>

    <key>RunAtLoad</key>
    <true/>

    <key>StartInterval</key>
    <integer>300</integer>
    
    <key>WorkingDirectory</key>
    <string>/Users/szymon/</string>
    
    <key>StandardOutPath</key>
    <string>/Users/szymon/Library/Logs/pl.skrajewski.restic_backup.out.log</string>

    <key>StandardErrorPath</key>
    <string>/Users/szymon/Library/Logs/pl.skrajewski.restic_backup.error.log</string>
</dict>
</plist>

Because I want to run this job as a user, I’m going to register it as an agent. All you have to do is to copy this file to the ~/Library/LaunchAgents/ directory and load the job using launchctl.

> cp pl.skrajewski.restic_backup.plist ~/Library/LaunchAgents
> launchctl load ~/Library/LaunchAgents/pl.skrajewski.restic_backup.plist

Once the job is loaded, the first backup will be started. Launchd also loads jobs when a user logs in. If you want to know how it works, you can read the The launchd Startup Process section in the documentation.

Full backup script

There is a full backup.sh script. It writes the output will to StandardOutPath and StandardErrorPath files. You can preview them using your terminal or using the Console application.

This file is also available as GitHub’s Gist.

#!/bin/bash

PID_FILE=~/.restic_backup.pid
TIMESTAMP_FILE=~/.restic_backup_timestamp

if [ -f "$PID_FILE" ]; then
  if ps -p $(cat $PID_FILE) > /dev/null; then
    echo $(date +"%Y-%m-%d %T") "File $PID_FILE exist. Probably backup is already in progress."
    exit 1
  else
    echo $(date +"%Y-%m-%d %T") "File $PID_FILE exist but process " $(cat $PID_FILE) " not found. Removing PID file."
    rm $PID_FILE
  fi
fi

if [ -f "$TIMESTAMP_FILE" ]; then
  time_run=$(cat "$TIMESTAMP_FILE")
  current_time=$(date +"%s")

  if [ "$current_time" -lt "$time_run" ]; then
    exit 2
  fi
fi

if [[ $(networksetup -getairportnetwork en0 | grep -E "Home-Network|Work-Network") == "" ]]; then
  echo $(date +"%Y-%m-%d %T") "Unsupported network."
  exit 3
fi

if [[ $(pmset -g ps | head -1) =~ "Battery" ]]; then
  echo $(date +"%Y-%m-%d %T") "Computer is not connected to the power source."
  exit 4
fi

echo $$ > $PID_FILE
echo $(date +"%Y-%m-%d %T") "Backup start"

export B2_ACCOUNT_ID=$(security find-generic-password -s backup-restic-b2-accound-id -w)
export B2_ACCOUNT_KEY=$(security find-generic-password -s backup-restic-b2-account-key -w)
export RESTIC_REPOSITORY=$(security find-generic-password -s backup-restic-repository -w)
export RESTIC_PASSWORD_COMMAND='security find-generic-password -s backup-restic-password-repository -w'

/usr/local/bin/restic backup --verbose -o b2.connections=20 --files-from ~/backup.txt 

echo $(date +"%Y-%m-%d %T") "Backup finished"
echo $(date -v +6H +"%s") > $TIMESTAMP_FILE

rm $PID_FILE

MacOS Catalina: Dealing with System Integrity Protection (SIP) and new permissions

Before you start
If you don’t want to use the extra binary to handle specific permissions, you can add your shell (e.g. bash or zsh) to Full Disk Access list in Security and Privacy panel.

After upgrading the system to macOS Catalina 10.15, my backup script stopped working. The error log had error operation not permitted:

scan: Open: open /Users/szymon/Desktop: operation not permitted
scan: Open: open /Users/szymon/Documents: operation not permitted
scan: Open: open /Users/szymon/Pictures/Photos.photoslibrary: operation not permitted

According to Catalina’s landing page, Apple improved the security of the system by using read-only volume for system data and adding more permissions that applications have to ask for.

Data protections.

Apps must now get your permission before directly accessing files in your Documents and Desktop folders, iCloud Drive, and external volumes, so you’re always in control of your data. And you’ll be prompted before any app can capture keyboard activity or a photo or video of your screen.

https://www.apple.com/macos/catalina/

Unfortunately, the script didn’t ask me for permission for any directory or functionality. I tried to add the Full Disk Access to the backup script, restic, bash, and terminal, but I still got the same outcome.

After reading some discussion on GitHub, I found out that wrapping script in the binary may help solve this issue. I don’t know the underlying mechanism yet, but when you run the binary, it asks the user for permission to requested resources.

I wrote a small console application in Swift that runs my backup script.

import Foundation
import os

let task = Process()

task.launchPath = "/bin/bash"
task.arguments = ["/Users/szymon/scripts/backup.sh"]

do{
    try task.run()
}
catch{
    os_log("error")
}

task.waitUntilExit()

There is a security problem: the intruder may replace the content of backup.sh and run the custom script to gain extra privileges.

After compilation, I got the binary backup_wrapper. I put it in the same place as my original backup script and I changed the path to this executable in pl.skrajewski.restic_backup.plist.

<key>Program</key>
<string>/Users/szymon/scripts/backup_wrapper</string>

When I run the binary, it asked me for permission to resources. I didn’t have to add Full Disk Access to the binary, restic nor bash.

backup_wrapper binary with the permission to Files and Folder.

backup_wrapper binary with the permission to Photos.

It’s still a lot of hassle to set up the whole backup process. If you work with bash scripts, there is a big chance that you’ll get in trouble with permissions. I would like to hear other opinions about this mechanism in Catalina and how to deal with it. If you already solve this issue in another way, please let me know.

Summary

As you can see, by crafting things for your needs you don’t have to start with something big. Start small, and change it in time. Add only things that you need to have, not everything that comes to your mind. Otherwise, you will end up with a bloated script with tons of unnecessary features.

I would implement this as a separate piece of software, written e.g. in JavaScript, using ready-to-use modules. But I didn’t. Things have to be simple, because simple things are cheap, easy to use and maintain.

There are some extra features I have implemented in my backup workflow but I didn’t include in the article:

Anybar integration to check if the backup is already in progress.
macOS notifications – I consider to get rid of them because the workflow is stable enough and I don’t need to have feedback if the backup has started.
Integration with healthchecks.io to get feedback if something is wrong with the workflow.

I recommend you to invest some time to set up an automatic backup workflow. It’s like the insurance – you may even don’t need it. But if something happens, it’s better to have it. And remember to check from time to time if your backup is consistent and restorable. Or maybe should we automate it as well?

Changelog

Update 2019-11-28:

I added information, how I deal with the new permission system aka “entitlement”. It complicated this set up process a bit because it requires to compile custom binary, that runs the backup script.

Update 2021-03-07:

As readers confirm, there is no need to run the backup script script using the extra binary wrapper if you grant Full Disk Access to the shell you use with the script (bash or zsh). I added a proper info to the article in the section regarding custom binary wrapper.

Featured photo by kropekkpl from Pixabay.

Write code for people

Wed, 02 Oct 2019 00:00:00 +0000

The first draft of this article was about the code-style. I wanted to show you why it is important and why we should care about it. But the longer I thought, the stronger conviction I gained, that the code-style – whatever it means – is only a way to achieve a much more important goal.

I’m a big advocate of well-quality code. Besides good architecture, I pay attention to style – consistent spacing and indentation, coherent naming convention and other rules that make the code visually better. But how to convince someone, who never took care of style before, that is it a good thing?

Why should you take care of code?

It’s not obvious but the code you produce is not for the computer. Machines don’t need a beautiful code. They don’t even need your fancy file structure or well-designed architecture. A compiler translates your work to the machine code that doesn’t resemble your handsome code at all.

So why should you care? Because you write for people, not machines. Machines can run the program like a flash, whereas people need to read the code line by line to imagine what it does and what would the outcome be.

The good code makes this process easy.

By focus on the people, you switch the mindset from “Does it work?” to “Is it easy to understand?”. The computer understands the crappiest code in the world as long as it’s valid. People need more time to connect the dots and see dependencies in the code. You may help them by reducing the cognitive load by using proper abstractions, expressive naming, and consistent style.

Even if you think that it doesn’t apply to you, you should care. If you conduct a project alone in a company, someone else may join you because of higher demands. If you leave your job, another developer takes over your work. How smooth this process will be?

Instead of asking “Does it work?”, ask “Is it easy to understand?” and change, until the answer is “yes”.

Still in doubt? A personal project you’re working on would be something bigger one day or would be stuck in your hard drive. It’s your decision. Even so, it’s something that you write for your future self. You may come back to this code in the future, so it should be easy to understand and make a pleasure to work with.

Code style is subjective, but it doesn’t matter. It’s everything about our customs and habits. Each team, company, and community has its perspective. Because you write for people, you will know how to do it right.

Take care of the people, who will work with your code. Help them understand what the code does. Treat it like the common good. You will see, how its quality improves in time. People will thank you someday.

Featured photo by Mimi Thian on Unsplash.

The hidden cost of self-hosted services

Wed, 18 Sep 2019 00:00:00 +0000

The Internet is teeming with services you can take advantage of. You can store your files in the cloud, share movies, organize photo collections, talk with friends, manage your projects and so on. It also means that your data is stored outside your computer and you don’t have full control over it.

You may use self-hosted counterparts of services. Instead of accepting vague privacy policies, you may set the rules. However, you are fully responsible for sustaining your data.

I used to consider self-hosted services as a privacy-respective and cheap alternatives of popular services. The former is true, however, the latter factor is often far-fetched.

The definition of cheap

If you limit yourself to thinking about costs only in the category of money, you may fall into the trap of hidden costs. Nowadays, servers are affordable, so money it’s not a real problem. It is even cheaper if you decide to host your services on-site, e.g. at home or in the office.

Money is the most visible cost, however, it’s not the only one. The second one is time. To sustain the self-hosted services you need to devote extra time to installation, solving appearing problems, maintenance and so on.

It’s fun as long as it’s just for fun

Hosting your photo gallery or file server may be both fun and useful, however, you should ask yourself what do you want to achieve.

If it’s only for learning purposes, you should reckon with potential data loss. Your services may be unreliable and insecure, due to the unstable internet connection, lack of updates and improper configuration. If you plan to incorporate the service you host into your workflow, you have to adapt it to production usage.

What do you do if service that you use is unavailable and you’re the only one who can fix it? Do you want to be user or maintainer?

The hidden cost of maintenance

People often forget that everything around us requires our attention. Self-hosted services are not different. Attention should be considered a valuable asset. If you devote time and attention to maintaining your self-hosted services, you neglect the other, more important things you have to do.

It’s worth to calculate the value of your time. If you know how much your hour costs, you make more accurate decisions. Isn’t better to pay a few bucks once a month and spend the recovered time on something more productive?

Do you need self-hosted solutions?

Always have a goal in mind. If you want to learn or test something, don’t use the service you host as the production one. Sooner or later, it will get you in trouble.

If you want to have a privacy-respective alternative of another service, calculate the actual costs and check if it’s worth to hassle. Don’t underestimate the needs of the proper hardware, regular backups, and updates.

Sometimes you enforce yourself on a specific self-hosted solution while there is a simpler alternative. Try to take a step back and look at the problem more broadly.

Do you need a file synchronization server but you don’t want to send your files somewhere? You may encrypt files before sending them to the cloud.

Do you need an online photo gallery to showcase your photos at family events? How often you need to do that? Isn’t a local gallery on your computer enough? How much time do you save hosted your gallery?

You may find more and more examples. A self-hosted solution should solve the problem you have, instead of adding other ones. Treat it as a rule of thumb. If something requires more resources than it gives you value, eliminate it.

A goal is rarely a “having self-hosted service”. The self-hosted solution is often one of possibilities to solve the actual problem. Choose the simpler one.

Featured photo by Gayatri Malhotra on Unsplash.

Collecting tiny thoughts

Wed, 21 Aug 2019 00:00:00 +0000

I collect lots of thoughts in my private notes. Some of them could be quite useful if I show them in a decent form. Unfortunately, they are too small to be a full article, so I decided to introduce another form of posts – tiny thoughts.

Tiny thoughts are thoughts that are too small to be an article but are too important to forget it.

I select only thoughts that are useful for you as well. They are my thoughts and my conclusions, so they are subjective. But I hope you will also find value in them.

It’s hard to predict what I’ll be writing about. I spend a lot of time learning, exploring and trying new things, improving myself and my life, so it would be an interesting topic. Nevertheless, I feel an urge to write a few words about things I do – either successful or failed ones.

Thank you for reading it. 😊

Featured photo by Dan Gold on Unsplash.

Use make as task runner

Wed, 07 Aug 2019 00:00:00 +0000

During development, you probably take advantage of some extra command-line tools. In PHP world it could be a mess detector or program to check the code style. The framework you use also exposes some functionalities to clear cache, migrate database or generate documentation. All these commands are helpful but you need to look for them until you memorize the most useful ones.

Sometimes you need to perform a task, like project initialization or restoring a stable snapshot of the database. It’s rarely an atomic operation, so you need to execute a few commands in a specific order.

This is a place where a task runner comes to play. Take a look at how a program called make can help you organize common tasks in your project.

Why make?

Make is a tool to generate files from other files, e.g. executables from the source code. But it can also perform other operations like running other tools and commands.

Unlike gulp or grunt, make doesn’t need any special JavaScript’s runtime or plugins. If you can describe your task using a set of shell commands, you may use them as your task in make as well.

Install make

macOS

If you use a macOS, you may install make using homebrew:

$ brew install make

Debian, Ubuntu

If you use a Debian-like system, it’s likely that you already have make. If not, you can install it from the repository:

$ sudo apt install build-essential

Windows

I have no experience with make on a Windows machine, but according to this answer from StackOverflow, you can install make using chocolatey package manager.

> choco install make

Define your first task

To define your first task, create a file called Makefile with the following content. Please be aware, Makefile uses tabs as indentation.

init:
    @echo "Initialize project"
    @echo "End of initialization"
.PHONY: init

Then, run the task by executing make init. The output should be like this:

$ make init
Initialize project
End of initialization

The Makefile is a bag for your tasks. Each task (called target) consists of commands that can be executed through the shell.

Because it’s a file, you can store and version it together with your project files in the VCS. Any team member may define other tasks to build up a set of tools and procedures for the whole team.

Thanks to the @ symbol at the beginning command, make displays output without printing the executed command. Whether it is useful or not depends on your use-case.

The extra line containing .PHONY: init means, that task doesn’t produce any file. It would be problematic if you have a file with the same name as the make target.

Make Make more useful

What is in your Makefile depends on your needs. Think about the goal of each task. Don’t put everything you suppose you will need. You may consider adding a new task if it:

adds any value for you and your team
speeds up your work
simplifies any complex operation

Here are a few examples of tasks from my Makefile.

## Initialize the application
init:
    docker-compose exec app bash -c "composer install -n -o --dev"
    make init-db
.PHONY: init
    
## Go to the shell inside app container
jump:
    @docker-compose exec app bash
.PHONY: jump
    
## Fix code style issues using phpcbf
cbf:
    ./vendor/bin/phpcbf --extensions=php --standard=phpcs.xml -p src/ --warning-severity=9 --cache
.PHONY: cbf

## Show code style issues using phpcs
cs:
    ./vendor/bin/phpcs --extensions=php --standard=phpcs.xml -p src/ --warning-severity=9 --cache
.PHONY: cs

## Migrate database
migrate:
    @docker-compose exec app bash -c "php ./bin/console doctrine:migrations:migrate"
.PHONY: migrate
    
## Initialize database    
init-db:
    @docker-compose exec db mysql -u root -proot -e "DROP DATABASE IF EXISTS app_db" > /dev/null
    @docker-compose exec db mysql -u root -proot -e "CREATE DATABASE app_db CHARACTER SET utf8 COLLATE utf8_unicode_ci" > /dev/null
    @make migrate
.PHONY: init-db

Creating a good Makefile is more like a process rather than a single action. Trying to define everything in advance is unproductive and may lead to creating many tasks you will rarely use. Start with something small and improve in time.

Let Make document itself

Since it’s worth to have a distinct name of the task, the goal of task runner is to save your time and memory. Instead of creating tasks with long names, you can define another task that displays the description of the task alongside its name.

GREEN  := $(shell tput -Txterm setaf 2)
YELLOW := $(shell tput -Txterm setaf 3)
WHITE  := $(shell tput -Txterm setaf 7)
RESET  := $(shell tput -Txterm sgr0)

TARGET_MAX_CHAR_NUM=15

.DEFAULT_GOAL := help

## Show this help message
help:
    @echo ''
    @echo 'Usage:'
    @echo '  ${YELLOW}make${RESET} ${GREEN}<target>${RESET}'
    @echo ''
    @echo 'Targets:'
    @awk '/^[a-zA-Z\-\_0-9]+:/ { \
        helpMessage = match(lastLine, /^## (.*)/); \
        if (helpMessage) { \
            helpCommand = substr($$1, 0, index($$1, ":")); \
            sub(/:/, "", helpCommand); \
            helpMessage = substr(lastLine, RSTART + 3, RLENGTH); \
            printf "  ${YELLOW}%-$(TARGET_MAX_CHAR_NUM)s${RESET} ${GREEN}%s${RESET}\n", helpCommand, helpMessage; \
        } \
    } \
    { lastLine = $$0 }' $(MAKEFILE_LIST)

Here’s a link to the source code. You may find other implementations on GitHub or StackOverflow.

Using my Makefile, it produces output like this:

$ make

Usage:
  make <target>

Targets:
  help            Show this help message
  init            Initialize the application
  jump            Go to the shell inside app container
  cbf             Fix code style issues using phpcbf
  cs              Show code style issues using phpcs
  migrate         Migrate database
  init-db         Initialize database

Which looks like

Self-documented Makefile – each task has the description

As you can see, the help message is displaying by default if you provide no arguments. It’s configured by setting .DEFAULT_GOAL := help.

If tasks’ names are longer than 15 characters, you can adjust the TARGET_MAX_CHAR_NUM variable.

Summary

Make is not the only tool in this area. Aforementioned gulp and grunt are also good but they need some extra resources and plugins. If you already work on JavaScript project, you can consider using either. Even so, it’s more likely that you already use a tool like webpack to bundle your application.

Using a task runner is great to hide complex operations behind simple commands. However, it’s also worth to know, what specific task does under the hood. The goal is to save time, not to reduce the necessary knowledge. In case of any error, who will be able to fix a bug, if the last modification to Makefile has done one year ago by a person who isn’t in the project anymore?

Featured photo by sergey Svechnikov on Unsplash.

5 stages of development as the programmer

Mon, 27 May 2019 00:00:00 +0000

5 years have passed since I had started working as a professional software developer. Although the programming was my hobby since the early years, my skill started to grow when I had taken a real job.

I was surprised when I had recalled my very beginning because I noticed how much I had changed and how big progress I had taken. So, I decided to write it down.

In this article, I’ll take you on a short journey where I’ll tell you about my development process. You’ll find out what I’ve been thinking about, what I’ve been working on and ultimately what changed from my perspective. Ready?

Junior developer writes working code

Everybody, who started programming knows, that the first of most enjoyable moments is when a program starts to work. It doesn’t matter it writes a static text on the screen, shows a message after button pressing or does a less ambitious job. It works.

At this stage, nobody thinks about architecture. At least I didn’t think. I was fascinated by what I can program yet. Bots to various games, simple website or even more advanced CMS systems. The more experimental projects, the better.

Over time, more advanced projects appear, and with them the first major problems. Problems with the code structure, duplication, readability, and other mess, that lead to increasing frustration. It takes a lot of time to introduce changes and the outcome is rarely satisfactory.

Source: giphy.com

The lack of knowledge about basic patterns can stop even the most ambitious project. The code becomes unmanageable. In the outcome, it’s easier to discard everything and start writing from scratch. Unfortunately, with the same effect.

Some smart people and interesting books have suggested what can I do with it. Making use of rules such as SOLID, DRY, and KISS was the first suggestion I read – I don’t even know where. It sounds great in theory. However, these rules are problematic even for more advanced programmers, let alone beginners.

The solution seems to be the use of design patterns. They will be my basic tool at work soon.

The use of design patterns

During discovering the programming world, I figured out, that most of the problems I had struggled with had been already solved. Some of them were generalized enough to fit into known design patterns, e.g. those proposed in the book of a gang of four.

The whole idea behind design patterns was encouraging. They provide battle-tested solutions for specific problems. For me, as a beginner programmer, it was a way to put a piece of code in a good place in a good way.

Overuse of design patterns

Design patterns were such a great tool, that it seemed almost a mistake to write the code without using them. In the outcome, after some time I saw the possibility of using a design pattern everywhere. Even where it was not needed.

At first, having a generalized solution to every problem seemed like a good idea. Most often, however, the problem became a cluster of abstractions. What he did was deeply hidden in the code.

Now I know that the design pattern is some kind of framework to solve the problem, but it is not a ready solution in itself.

In this time, I’ve been writing most of the solutions by myself. I didn’t want to use ready-made solutions such as frameworks or libraries. On the one hand, it was good because I learned and understood a lot of things when I had to implement them. On the other hand, it required much more time, but the results were rarely satisfactory.

The design pattern is a generalization of the solution, rather than the solution itself. It provides some kind of framework to solve a specific problem.

When I got bored with writing the same things over and over again, my attention switched towards ready-made solutions.

If there is a problem, there is also a library that solves it

I joined in the framework-world quickly. They have added extra restrictions, especially when it comes to the code writing. In the result, they made the work more structured. Frameworks also have had a lot of components that increase the speed of development.

If there is a problem, it’s very likely that there is also a library that solves it. I got used to this approach very fast.

This chart shows growth of quantity of libraries and their versions in the packagist.org registry. There is above 225 thousands of libraries in May 2019.

Focus on ready-made libraries instead of solutions

The confidence that there is a library for every problem got me into troubles. Instead of focusing on solving problems, I spent a lot of time looking for a library which would solve it for me. I usually found what I had been looking for and finally, I solved the problem. It doesn’t mean that everything was seamless. Most of the time, third-party code put extra restrictions, which turned into other problems, especially in some more specific business cases.

After some time I started looking very carefully to each library I would incorporate into my project. If I need to spend extra time to adjust functionality to requirements, maybe it’s worth to write it from scratch? In time, more question emerged, e.g. if I use another library, will I have fewer or more problems? Does it solve my problem?

Adjusting ready-made solutions according to client’s requirements. Source: giphy.com

Problem solving

At some point in my career, I understood, that solving problems is not about writing software at all costs. Some classes of problems could be solved by existing software. The key is to know what could be used.

Listening to the client’s needs

I’ll give you an example. Let’s assume that you receive a request about making an application to generate summaries and reports from the company. The client, however, has a very limited budget. You’re not so interested in this cooperation but for some reason, the client reached you or your company.

This should lead you to a few questions, that you should answer.

Why did the client get to me? Do I aim right customer group?
Since this client has already asked me, how can I help him?
- Does the client need a dedicated software?
- If he doesn’t need dedicated software, is any other tool that I could offer him instead? E.g. Microsoft Excel, Calc from Libre Office or any other spreadsheet?
- Could I recommend the client to another company specialized in problems that the client needs to resolve?
- Could I do anything for the client in his budget?

As you can see, not everything has to come down to writing code. If it does, it should take care of solving an actual client’s problem to meet his needs.

Architecture, paradigm, language, framework are only tools, which have the one important task. They are to help in solving a domain problem.

You’ve noticed, that I write lots about problems. I don’t mean the challenges we set for ourselves as programmers by choosing architecture, paradigm or framework. They are our tools, that are designed to help solve the higher level problem, usually a business problem or more generally – a domain problem.

Solving domain problems

Domain problems are the most interesting because they need more than programming knowledge. You need also a specialized domain knowledge. They’re real challenges, that require from the programmer constant improvement and willingness to learn new things, which are not necessarily related to the IT world.

There is where the famous DDD acronym – Domain-Driven Design – appears. It is a holy Graal to every programmer aspiring to be an architect or a domain designer. It’s a place, where developers know what the business expects and, at the same time, the business feels that developers understand its needs.

To make it possible, you should discard all things that introduce unnecessary noise. You may use them only where it’s necessary to solve the problem. You should simplify complicated things, and you should make simple things even simpler.

The favoring of simplicity drives to the situation, where a developer starts to write a code that just works. And it’s simple.

Senior developer writes working code

I’m not looking for projects which I could do in my favorite framework or with the usage of currently top-fashioned architecture. I don’t try to replace each loop by an iterator and I don’t try to complicate things that can be simple.

I don’t reinvent the wheel and I use tools which are designed to solve specific problems. Each written line of code is something that needs maintaining in the future.

Instead, I help others in solving their problems. I try to understand what they need and I suggest the best solution to a specific problem and situation.

Moreover, I also take advantage of the generosity of other programmers who share their knowledge and work. I try to do the same by showing how to write good code and by sharing my knowledge and experience as well.

When I write code, I know, that I’m not the only one who will be reading it. Many eyes will be reading and trying to understand how given functionality was implemented and what it does. My job is to help others understand this.

All these things combined with my experience are my workshop. Thanks to which I can effectively solve further problems, constantly improving myself. Every day I get to know new things and I decide, which of them I can include in my workshop.

Summary

I share my thoughts because I want to show you how at different stages of my life and career I have changed my perspective and my way of thinking.

Once I tried to write as much as I could by myself. Another time I was looking for ready-made solutions. Then I tried hard to use what I had learned. It took me years to learn how to do it well.

As you can see, a lot of things changed – in my opinion for the better. Changes are visible consequences and proof of making progress. However, they are not the result only of the passing time.

Changing a job, other people, maintaining healthy relationships, environment, study, and books. They are some of the most important elements that have influenced my way of thinking. That’s why is so important to open yourself to new experiences and confront your thoughts with new ideas.

What I have described here is my journey, which doesn’t necessarily coincide with yours and probably will not. However, if you somehow identify with what I have presented, please tell me about it by leaving a comment or by e-mail.

This isn’t the end of the journey.

Before we say goodbye, I would like to leave you with one thought – What is the most important change you see in yourself, in comparison to yourself from a year ago?

Featured photo by Émile Perron on Unsplash.

Automatic file versioning after change using fswatch and git

Mon, 01 Apr 2019 00:00:00 +0000

I create a lot of notes. Seriously, I write tons of notes. Inspired by Getting Things Done method, I treat my mind as a thoughts generator rather than the storage. I capture thoughts, ideas, inspiring quotes, links, and pictures. Mostly using files.

Although I’m on the early stage of creating my custom note-taking solution, I’ve done some work to synchronize notes between devices and versioning them. In this article, I’d like to focus on the latter and I’m going to show you how to set up automatic files versioning.

I’m a software developer so, unsurprisingly, I use git to versioning my notes as well. Instead of manually committing changes, I take advantage of tools like fswatch and launchd to automate this process.

The problem

My notebook (if it’s a proper name for a directory with files) consists of simple text and Markdown files as well as attachments to them like pictures, archives, records, etc. The internal structure doesn’t matter.

Keeping notes as regular files instead of dedicated Notebooks (like Evernote or OneNote) let me use any tool I need at the moment or I really like. It has also some drawbacks – most of the features provided by dedicated software are missing, including the synchronization and versioning of my notes.

I decide to use git to keep track of changes in files. Moreover, I want to automate this process. I don’t want to remember to commit each change in the note, so I delegate this operation to my OS.

Initial steps

Prepare repository

Make sure that you have git installed on your OS. I use macOS, the fastest way to install any software is using homebrew.

brew install git

Then, create a dedicated directory for your notes/files and put them here. I store my versioned notebook under the path /Users/szymon/Documents/Notes.

It’s time to initialize a new repository. Using terminal, go to the newly created directory and run git init.

cd /Users/szymon/Documents/Notes
git init

The repository is ready. It’s time to configure watcher to receive information about changes in files.

Install fswatch

Actually, we don’t need to know what exactly has changed (it will be covered by git). We need to know only that change occurred in our repository.

I choose fswatch as a tool to observe changes in the directory. It’s cross-platform, works well on macOS and is able to check directories recursively. Launchd, however, is also able to watch for changes in the directory but it doesn’t track nested directories.

You can install fswatch using brew.

brew install fswatch

If it’s done, we can go to connecting these tools.

Prepare scripts to operate

One of the UNIX philosophy sounds “Make each program do one thing well”. Actually, UNIX-like systems are full of programs which do one thing: cat, grep, tail, head, xargs. The true power lies in the possibilities of connecting all these little programs together.

Let me describe what we need to achieve. For each change in the directory we need to add changed files to stage and commit them. So, we have basically 2 different operations which are connected:

(1) Watch the directory and run command for each change
(2) Stage and commit all files in the repository with message

Creating these 2 scripts separately gives us more possibilities. We can test them easily, we can replace one of the scripts with another one, we can reuse them in a completely different scenario, etc.

Add all to the stage and commit

First, let’s create the script to stage and commit all changes in the repository (2). I usually put scripts like this in ~/.scripts directory. You can create this file using the command touch ~/.scripts/commit-all.sh.

#!/bin/bash

git add .
git commit -m "Changed: $1"

You can also add git push command to push changes to the remote repository.

If you’re more advanced git user, you probably know, that git allows to pass option -a to commit command to stage modified files. Unfortunately, it ignores new files which are not tracked yet.

-a, -all Tell the command to automatically stage files that have been modified and deleted, but new files you have not told Git about are not affected.

The $1 is an argument to the script and it’s used in the commit message. Make this script executable by command chmod +x ~/.scripts/commit-all.sh.

Run command to observe changes

The second script runs fswatch with a set of options and passes the name of the changed file to the further execution. Start with creating script touch ~/.scripts/run-on-change.sh.

#!/bin/bash

/usr/local/bin/fswatch -e ".git" -e ".*\.swp" -e ".*~" -0 . | xargs -0 -n 1 -I {} basename {}

Similarly, we need to make this script executable by command chmod +x ~/.scripts/run-on-change.sh.

To prevent watching specific pattern I exclude them using the -e option. It’s important to exclude the .git directory – out intention is to automatically commit all detected changes. We don’t want to commit anything after changing something in the repository.

Combine scripts

We have a simple script which detects changes in the current working directory and another one which adds and commits all files to the repository. Let’s combine them together.

In your repository, run following command

~/.scripts/watch-and-commit.sh | xargs -I{}  ~/.scripts/commit-all.sh {}

Now, if we change or add a file to the directory, it would be immediately committed. Moreover, the commit message would contain the name of the changed file.

Schedule execution using launchd

It’s not the first time when I use launchd as a job scheduler instead of a cron service. I briefly wrote about it in my article about synchronizing local files with encrypted vault. Basically, launchd is a recommended solution for macOS for job scheduling.

Because I use macOS on a daily basis, I focus mostly on this platform, however, we can reach the same goal using a tool called supervisord. It’s my first choice when I will configure this flow server-side or on Linux-powered machines.

I created a pl.skrajewski.notes.autocommit.plist file in directory ~/Library/LaunchAgents/ which is one of the locations for agents which work on behalf of the user.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN http://www.apple.com/DTDs/PropertyList-1.0.dtd>
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>pl.skrajewski.notes.autocommit</string>
  
    <key>ProgramArguments</key>
    <array>
        <string>bash</string>
        <string>-c</string>
        <string>~/.scripts/watch-and-commit.sh | xargs -I{}  ~/.scripts/commit-all.sh {}</string>
    </array>
  
    <key>KeepAlive</key>
    <true/>
  
    <key>WorkingDirectory</key>
    <string>/Users/szymon/Documents/Notes</string>
  
    <key>StandardOutPath</key>
    <string>/tmp/pl.skrajewski.notes.autocommit.out.log</string>
  
    <key>StandardErrorPath</key>
    <string>/tmp/pl.skrajewski.notes.autocommit.error.log</string>
</dict>
</plist>

In short: Label is an identifier of the agent. System runs the command specified in ProgramArguments in WorkingDirectory and redirects stdout and stderr to files defined as StandardOutPath and StandardErrorPath. The option KeepAlive means, that launchd will restart the job each time it goes down.

The job will start once is loaded. You can do this using launchctl.

launchctl load ~/Library/LaunchAgents/pl.skrajewski.notes.autocommit.plist

If everything is set up properly, each change in directory produces new commit. If not, we can use log and error file to investigate what goes wrong.

Weak points of this solution

This solution is far from ideal and has some weak points.

A proposed watcher, fswatch doesn’t observe changes in .git directory due to exclusion policy. However, if we checkout past commit, it will be treated as a change, so the script will be trying to commit changes. If you want to restore the previous version of files, make sure that the agent is not running. You can unload it using the following command.

launchctl unload ~/Library/LaunchAgents/pl.skrajewski.notes.autocommit.plist

Nevertheless, it’s better to change the script a bit to avoid committing if e.g. HEAD is from master branch. It’s a field to improve.

The problems may come also from the editor you use. Some of the editors store temporary files next to original files and they may trigger auto-commit each time when the content of file change. You’ll have to exclude these files in the fswatch invocation.

Additionally, a lot of editors have auto-save functionality, creating in outcome lots of commits with relatively small differences. Instead of committing changes immediately, we can delay this operation a bit. If something change before the commit, the script will cancel the previous invocation and it’ll schedule the commit with new changes.

Summary

Although the whole process sounds like reinventing the wheel, I hope that you found out something interesting for you. You can treat this article like a proof-of-concept or a presentation of the idea rather than the complete recipe.

I used to write software which does something I needed. Sometimes, I spent days or weeks before I got something useful. Recently, I’ve discovered the real power of combining small and simple tools, mostly provided by my operating system. It just works.

It doesn’t mean that writing software is bad. In my opinion, writing is the best way to learn. But the main goal of software should be to solve a problem. If the problem can be solved using tools we already have, why not use them? At least until we want to generalize the use case to be useful for others.

Featured photo by Ag Ku from Pixabay.

How to use final in PHP?

Mon, 18 Mar 2019 00:00:00 +0000

If someone would ask me about my favorite keyword in PHP, I would certainly answer: final. It doesn’t mean I write this modifier in each class or method. It not only shows the intention but also provides a mechanism to protect the code. At least from the assumption.

The final keyword can be used both on the class level and on the method level. It prevents future extension of functionalities in a non-effective way. If a class is marked as final, then we can’t inherit from it. If a method is marked as final, we can’t override it.

The theory sounds good. Let’s go to details.

The intention behind final

In object-oriented programming, you can extend the behavior of the class by inheritance. It’s simple and convenient.

Do you remember the example classes Figure and its children: Square, Rectangle, Polygon, etc.? It’s a classic example of inheritance and polymorphism in work. In this case, you probably don’t want to finalize Figure class.

The final keyword added to the Figure class puts the extra restriction on the subject, so inheritance is no longer possible. So why does this mechanism even exist? Who would like to limit the usability of own solutions?

But finalizing is not the same as limiting.

Imagine the hypothetical class ArticleProvider whose only one public method getAll() is responsible for returning collection of articles, e.g. from my blog. Moreover, let’s say, that this class fulfills the interface called ArticleProviderInterface.

If you want to operate over my articles, you have two possible ways. You can extend the class ArticleProvider using inheritance or you can use the class, e.g. using composition. Both ways lead to a proper solution, but because I’m the author of the class, I would like to encourage you to choose the safer option – composition. It prevents creating a chain of inheritance and preserves the encapsulation of behavior and state in class.

Since there is no special reason why someone would need to extend ArticleProvider, I decided to mark them as final. It means the class is ready to use as one of the implementations of the interface ArticleProviderInterface. Other developers may utilize only the public API of this class.

Final as the synonym of “ready”

An interface is a contract between developers and people who will be using the class so they can be sure that the behavior of methods specified by interface will not change.

Final, however, is a contract between developers to make sure, that the implementation of the specific class will not change in the future.

The class marked as final means, that the specific implementation of the interface is final. In other words, it’s complete in the sense, that there is no need to add any other functionalities there.

It looks like a good moment for an example. I’m going to create a simple mailer, so let’s define an interface for it.

interface MailerInterface
{
    public function send(string $email, Message $message): bool;
}

I need at least two implementations of this interface: the first one should actually send an email. For this example, we can use a PHP’s builtin mail function. The second one should write messages to file – I’ll use it in tests.

final class BasicMailer implements MailerInterface
{
    public function send(string $email, Message $message): bool
    {
        return mail($email, $message->getSubject(), $message->getBody());
    }
}

I think, that above class is functionally complete and fulfill the interface, so I decided to mark it as final. The second implementation is as follows:

final class FileMailer implements MailerInterface
{
    public function __construct(string $filePath)
    {
        $this->file = $filePath;
    }
    
    public function send(string $email, Message $message): bool
    {
        $payload = json_encode([
            'email' => $email,
            'subject' => $message->getSubject(),
            'body' => $message->getBody()
        ]);
        
        return file_put_contents($this->file, $payload) !== false;
    }
}

As previous, I marked this implementation as final. Nobody should expect more from this simple class – it does exactly what it should.

This example is suited to be used with the final keyword, but most of the classes we write aren’t. It’s important to know when we want to use this feature. I write want because there are no “should” or “should not”. You, as the developer and author, decide how your code is flexible. I’m going to write about this later.

How to introduce change if a class is final?

First, answer this question: “Does the change you would like to introduce isn’t the other implementation of specific interface?”

Consider this scenario: Our platform serves users’ avatars from our own FTP server. Due to cost-reduction, business would like to migrate to the Amazon S3. Avatars are accessible by AvatarService which fulfill the interface AvatarProviderInterface.

interface AvatarProviderInterface
{
    public function getForUserId(int $id): string;
}

Although it’s not explicitly marked, AvatarService is final in that sense, that it provides access to avatars from FTP server. Instead of changing its implementation to S3, we should create another implementation of AvatarProviderInterface, e.g. S3AvatarService.

We finished with 2 completely different implementations of the same interface. Both are in some sense final – we don’t expect anyone to extend them. Moreover, in any moment, we can switch to the previous implementation. We may also deprecate it, if is no longer needed.

I recommend asking yourself if the change isn’t the other implementation each time when we need to change something in the class, even for a non-final class.

If it’s a bug that needs to be fixed in existing implementation, don’t worry. Go to your tests and spend a few minutes writing a missing scenario that covers this issue. Then, make tests green again.

Tests with final class

Tests are always connected to specific implementations. There is nothing wrong to mock a filesystem, an external API or a queue to check if specific implementation works properly.

I often hear an opinion that trying to cover each line of methods is sometimes like concreting the implementation. Whatever you change in the code, the covering test has started to fail.

Instead of identifying another implementation, developers are trying to change existing and completed (final) classes. Of course, there are exceptions. Nevertheless, it’s normal, that if you change the implementation, you have to also change the way how you test it.

How to mock final class?

One of the arguments against using the final keyword in classes is that we’re unable to create a mock of this class in order to test the dependent service.

It might sound strange, but this behavior is completely O.K.

Actually, you should mock an interface (generally speaking, abstraction) rather than the implementation of it. If you follow this rule, the problem with mocking final class will disappear. Unless you use methods or properties from outside the interface. I wrote why we shouldn’t do that in other article.

It shows also another concern: not everything should be marked as final.

When to use a final modifier?

There is no better than Marco Pivetta’s answer:

Make your classes always final, if they implement an interface, and no other public methods are defined

For any other situation, I would consider rethinking an approach a bit and check if a class doesn’t break any of SOLID principles. It may turn out, that making the class as final is ineffective, e.g. for classes like DTOs, controllers – a class without interface – or entities, (e.g. Doctrine’s entities) where final breaks proxy mechanism.

Summary

Marking class as final is a contract between developers, that the implementation of the class will not change in the future. Interface, on the other side, is a contract which ensures that class has specific behavior.

The final modifier may apply to both classes or methods. It prevents further modifications by inheritance, encouraging developers to use composition instead.

I recommend creating classes with the intention to “finalize” them. It’s not an obligation and you may remove this modifier anytime. It’s easier to transit from final to non-final class rather than the opposite direction.

Give yourself time to check if you’re comfortable with this approach. You’re going towards a better, cleaner code.

So, at the end I have one thought: If abstract means, that implementation is not ready, shouldn’t the final be a mark of its completeness? I’ll be happy to hear your opinions.

Featured photo by Angelina Litvin on Unsplash.

The practical difference between "findBy" and "getBy" in repositories

Mon, 25 Feb 2019 00:00:00 +0000

Repositories are a special example of a class. They usually have a lot of methods designed to retrieve data from the database or the other storage. To mark this operation in the name of the method, we can use one of the common words: find, get, search. Are all them mean the same? In this article, I would like to show you a practical difference between getById and findById methods.

Find vs Get

It’s nothing wrong to use find and get words in methods’ name interchangeably. Both of them have similar meanings and mark the intention properly. However, if we decided to differentiate them, we’ll get an extra benefit as the more meaningful and consistent code. Let me give you an example. Please see at the following:

interface ArticleRepository
{
    public function getById(int $id): Article;
    
    public function findById(int $id): ?Article;
}

Did you pay attention to the extra question mark in the second method’s header? If you already know what’s going on, you can stop reading here. You can also leave a comment under this article describing why you use this convention. Otherwise, you can continue reading.

As you can see, the only difference (besides the name of the methods) is the return type. This is related to the error handling mechanism — what should be done if the entity doesn’t exist?

Return null or throw an exception

First method, getById always returns an Article object. You may use this method in a workflow if you’re sure that the specific entity exists. E.g. when you have checked condition related to its existence. If for some reason the entity doesn’t exist, there is an actual unexpected situation and the method throws the exception.

If we could embody the getById method, it would be a kid who always gets what he wants. Otherwise, he will scream like a crazy. You should know beforehand how to deal with him.

On the other hand, findById either returns Article or null. It depends on the existence of the requested entity. You may use this method if you’re not sure if this entity actually exists, e.g. when someone requests a specific article through an API.

In this case, the findById method is more like a customer in a grocery store. He may ask for something and get it. Another time, he may ask for a blue tomato and get nothing. It happens.

Small improvements, big results

Although this proposal isn’t revealing and ground-breaking it makes the code more predictable and consistent. The more predictable and consistent code, the easier to read and understand it is.

Remember, that nothing works separately. If you want to improve the codebase of your project, you have to introduce a set of easy and useful standards and techniques. If you know polish, I encourage you to read my article “How to update standards in the aged project” 🇵🇱.

In our team, we successfully use this convention, sometimes in conjunction with return early approach. Our controller’s actions often have to return details regarding the requested entity. If the entity doesn’t exist, we handle it immediately. For instance:

public function actionShowArticle(int $id): Response
{
    $article = $this->repository->findById($id);
    
    if ($article === null) {
        return $this->errorResponse(StatusCode::NOT_FOUND);
    }
    
    return $this->response($article);
}

Or the other example, if you prefer exceptions:

public function actionShowArticle(int $id): Response
{
    try {
        $article = $this->repository->getById($id);
    } catch (EntityNotFoundException $e) {
        return $this->errorResponse(StatusCode::NOT_FOUND);
    }

    return $this->response($article);
}

The most important is to keep consistency in the behavior of findBy… and getBy… methods. The inconsistent code is untrustworthy, so each time, developers have to make sure that the method is implemented properly, instead of taking it from granted.

Summary

It’s all about the convention and standards. These things are useful as long as there are people who want to use them. If you feel, that this tip might be useful in your project, don’t hesitate to discuss this idea with your team.

Even in an old project, it’s still worth to propose and introduce improvements, to reduce a cognitive load of the code. The work with this code will be much easier and more enjoyable, for you and your teammates.

If you like this article, I’ll be grateful if you share it further. Thank you!

Featured photo by Aaron Burden on Unsplash..

Quick start with TDD in JavaScript using Jest

Mon, 22 Oct 2018 00:00:00 +0000

JavaScript is one of the most versatile programming languages I know, however, I had a constant problem with the testing. Since I run into TDD, I was looking for something that allows me to build my solutions using this approach. When I had been creating a small project in React, I accidentally discovered Jest.

It surprised me because it hadn’t required any extra configuration. I thought that the configuration was built-in into the create-react-app tool. It turns out that zero-configuration is one of the philosophies behind this tool. I decided to try using it outside the React application. Since this moment, this is a standard tool for my unit tests for the JavaScript.

Let me show how easy work with Jest is.

How to start testing with Jest

Add Jest to the project

First, we need to install Jest as a dependency. We can also install it globally but I prefer to treat it like a normal dependency within the project. The easiest way to do it is to use yarn or npm.

yarn add --dev jest

The next step is adding a starting script to the package.json file to run the test using locally installed instance.

{
  "scripts": {
    "test": "jest"
  }
}

Run tests in the watch mode

If we’re working in the project using git, Jest uses the data from the repository to identify and runs tests for modified files. It’s necessary if we want to run it in the watch mode.

yarn test --watch

Otherwise, if we’re not using any version control system, Jest isn’t able to identify changed files. We can still watch for changes in every file using the —watchAll flag. It’s OK for the small number of files.

yarn test --watchAll

Of course, we can still narrow the watching scope to a specific name or path to file, but let’s keep things simple for the beginning. For more information, take a look at the documentation.

Configure Jest to work with Babel

I cannot imagine work without the ES6 syntax in the JavaScript. It introduced lots of useful features which make JS easier to use. To support Babel, we need to add a few more dependencies.

yarn add --dev babel-jest babel-core babel-preset-env regenerator-runtime

According to the documentation, regenerator-runtime isn’t needed if you use yarn or npm (v3 or v4).

You can also have to adjust the .babelrc file:

{
  "presets": ["env"]
}

Write test

Now, we can switch to the writing the test. I prefer simple solutions, so I started with creating a test file in the same directory as the tested code. Let’s consider the function which normalizes an array of numbers.

// Normalizer.js
import max from 'lodash/max';
import min from 'lodash/min';

const scaler = (dataset, _min = null, _max = null) => {
    _min = _min || min(dataset);
    _max = _max || max(dataset);

    if (dataset.length === 1) {
        _min = 0;
    }

    return dataset.map(normalize(_min, _max));
};

const normalize = (min, max) => (val) => (
    (val - min) / (max - min)
);

export default scaler;

To start, we have to import the function in the scope of the test and check outputs for given inputs. Of course, we consider only unit tests for now.

// Normalizer.test.js
import scaler from './Normalizer';

it('should return an empty result for empty dataset', () => {
    expect(scaler([])).toHaveLength(0);
});

it('should return the same dataset for one identity element', () => {
    expect(scaler([1])).toMatchObject([1]);
});

it('should return the same dataset for already normalized elements', () => {
    expect(scaler([0, 1])).toMatchObject([0, 1]);
});

it('should normalize single element in dataset using it as a max', () => {
    expect(scaler([5])).toMatchObject([1]);
});

it('should normalize dataset using min and max from the dataset', () => {
    const scaled = scaler([0, 5, 10]);
    expect(scaled).toMatchObject([0, 0.5, 1]);
});

it('should normalize dataset using defined min and max', () => {
    const scaled = scaler([1, 2, 3, 4, 5], 1, 11);
    expect(scaled).toMatchObject([0, 0.1, 0.2, 0.3, 0.4]);
});

Thanks to the watching mode, we can receive the feedback after each change in both the test and the code. That’s great when you work using a TDD approach. Moreover, if you using an editor like Visual Studio Code or Atom, there are great extensions providing useful visual feedback directly in the editor (see resources).

Visual Studio Code with runner plugin.

Summary

This presented use-case is small, however, I often need a develop some small functionalities to confirm my initial idea. That’s why I need a tool, that supports me with my work instead of overwhelming me by its own configuration. By using Jest, I can focus on things I want to do instead of doing things I have to do.

I think It’s always worth to try new development tools. Sometimes a little change may have a significant difference in future work. But don’t take things for granted and test by our own. Maybe in your case, a different tool is much better? Let me know in comments.

Resources

https://jestjs.io/en/ – Jest documentation
https://babeljs.io/docs/en/ – Babel documentation
https://jestjs.io/docs/en/getting-started#using-babel – Using Jest with Babel
https://marketplace.visualstudio.com/items?itemName=Orta.vscode-jest – Jest runner for VSCode
https://atom.io/packages/tester-jest – Jest runner for Atom

Featured photo by Chris Liverani on Unsplash.

Set up multi-container environment using Docker Compose

Mon, 08 Oct 2018 00:00:00 +0000

I remember this time when I discovered Vagrant. The magic behind this tool and the general idea of scaffolding the whole environment using single command was pretty genius. Moreover, I had programmed on Windows and thankfully I could get rid of the XAMP and any other Windows-oriented web server packages.

In my previous job, my boss showed me a tool called Docker. I instantaneously got the point of the concept behind it and I started exploring the big universe of the possibilities of use. In this article, I’m going to show the most common use-case for docker – a proposed local environment suited for PHP application development.

This article is devoted to the configuration rather than explanation what the Docker really is. I would like to show the actual use-case rather than showing my point of view – I’ll follow it in one of the next articles.

Requirements

If you want to follow this instruction, you have to install:

Docker CE – docker engine
Docker Compose – tool for orchestrating the multi-container environment
PHP and Composer

If you don’t want to follow this article, don’t worry. Everything is prepared and ready to use – just see my GitHub repository skrajewski/php-app-on-docker on a tag basic-php.

Test application

There is no much sense to create an advanced environment to develop a single-file script. To approach ourselves to the actual situation we’re going to run a Symfony 4 application.

Our application should use PHP 7.2 and should be served by the nginx web server. Moreover, we need also a database to store our data, e.g. MariaDB.

Think about the infrastructural services

All these three services may be considered as infrastructural services. Each of them may live in a dedicated container – we don’t have to mix up the PHP with the web server.

This is our goal. Create a multi-container environment where services will be connected together if needed.

Install Symfony’s website skeleton

In this step, I assume that we have installed and configured PHP and Composer. Of course, we may start from the environment and subsequently run the whole set up within it. In this article, we’ll focus on putting the application into the new environment.

To install Symfony application, run below command

composer create-project symfony/website-skeleton application

Change the working directory to application and let’s go to the environment definition.

Development environment

We should prepare three different containers for each of our services. We don’t want to do it by hand. It’s the place where docker-compose come to the play. By preparing docker-compose.yml file we can describe how our environment should look like.

Create a docker-compose.yml with the below content

version: '3'

services:
    php-fpm:
        image: php:7.2-fpm
    nginx:
        image: nginx:1.11
    db:
        image: mariadb:10

However this config will create a three containers environment, there is a lot of missing configuration yet.

Make services accessible from the host

To reach each of this service we have to know their internal IPs. Unfortunately, they are dynamically assigned so it’s hard to simply prepare a host file. To mitigate this problem we can bind exposed ports to our host machine. Let’s do this!

version: '3'

services:
    php-fpm:
        image: php:7.2-fpm
    nginx:
        image: nginx:1.11
        ports:
            - "8080:80"
    db:
        image: mariadb:10
        ports:
            - "33060:3306"

The syntax is following: "${HOST_PORT}:{SERVICE_PORT}". Now, the web server should be accessible under localhost:8080 and the database under localhost:33060. We don’t need a direct access to php container.

Provide credentials to our database

In order to access our database, we need to set up a password at least for the root user. The documentation of MariaDB image on Docker Hub is pretty good and it shows how to do it using environment variables. Let’s change a docker-compose.yml configuration a little bit.

version:'3'

services:
    php-fpm:
        image: php:7.2-fpm
    nginx:
        image: nginx:1.11
        ports:
            - "8080:80"
    db:
        image: mariadb:10
        ports:
            - "33060:3306"
        environment:
            - MYSQL_ROOT_PASSWORD=root
            - MYSQL_DATABASE=app

Our database server will create a default database called app and set the root password’s to root. Not so secure, but it’s enough for the local development. Keep it simple.

Add a virtual host to nginx container

However nginx is working correctly, it shows nothing interesting than the standard welcome page. We have to add the configuration to let web server knows how to manage request to our application.

Personally, I like to keep all docker related stuff in the directory called .docker which I create in the root of the project. For a better order, I’m going to add subdirectory related to the service to keep all files together.

mkdir -p .docker/nginx
touch .docker/nginx/default.conf

Put the virtual host configuration to the newly created default.conf file.

server {
    # This is the root of our application, we need to mount files in /app
    root /app/public;
    index index.php index.html;
    server_name _;

    location / {
        try_files $uri /index.php$is_args$args;
    }

    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    location ~ ^/index\.php(/|$) {
        # Instead of using socket, we pass execution of PHP files 
        # to the process on different container using host name
        fastcgi_pass php-fpm:9000;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        fastcgi_param DOCUMENT_ROOT $realpath_root;
        internal;
    }

    location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
        expires max;
        log_not_found off;
    }

    location ~ \.php$ {
        return 404;
    }
	
    error_log /var/log/nginx/project_error.log;
    access_log /var/log/nginx/project_access.log;
}

As we can see in the configuration file, we use a php-fpm located in another container. This approach allows us to e.g. change the PHP version without changes in the web-server.

To make it works, we need to mount both configuration and application files into the nginx container.

version: '3'

services:
    php-fpm:
        image: php:7.2-fpm
    nginx:
        image: nginx:1.11
        ports:
            - "8080:80"
        volumes:
            - ./:/app
            - ./.docker/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
    db:
        image: mariadb:10
        ports:
            - "33060:3306"
        environment:
            - MYSQL_ROOT_PASSWORD=root
            - MYSQL_DATABASE=app

Mount the application to the PHP container

In spite of the nginx has access to the application files, the php-fpm container doesn’t know anything about the files. The php-fpm should also have access to application files in order to process them on the request of web-server. Let’s mount the application directory also to this container.

version: '3'

services:
    php-fpm:
        image: php:7.2-fpm
        volumes:
            - ./:/app
    nginx:
        image: nginx:1.11
        ports:
            - "8080:80"
        volumes:
            - ./:/app
            - ./.docker/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
    db:
        image: mariadb:10
        ports:
            - "33060:3306"
        environment:
            - MYSQL_ROOT_PASSWORD=root
            - MYSQL_DATABASE=app

In the old good days, docker-compose in version ‘2’ has option volumes_from which allows mounting volumes from the other container. It let us avoiding repetition but unfortunately, for various reasons, this option is no longer supported.

Use our own PHP configuration and extensions

Using the raw PHP image is very limited. The better option is to create own image using the PHP image as a foundation. We can do this by creating a new Dockerfile.

Similar to nginx configuration, let’s put the new file in .docker/php/Dockerfile.

mkdir -p .docker/php
touch .docker/php/Dockerfile

Now, edit newly created Dockerfile and put below content.

FROM php:7.2-fpm

RUN apt-get update && apt-get install -y \
    libmcrypt-dev \
    zlib1g-dev \
    libicu-dev \
    g++

RUN docker-php-ext-install -j$(nproc) iconv mysqli pdo_mysql mbstring intl

RUN yes | pecl install xdebug-2.6.1 mcrypt-1.0.1 \
    && docker-php-ext-enable xdebug mcrypt

Based on php:7.2-fpm we create a new image with some extra extensions, which are needed for our application. If you need more information about the php image, please check the documentation on the Docker Hub. The same thing we may do with the php.ini file, but we skip it for the sake of brevity.

The only thing to do is to use this image in our docker-compose.yml file. Let’s do it.

version: '3'

services:
    php-fpm:
        build:
            context: .
            dockerfile: .docker/php/Dockerfile
        volumes:
            - ./:/app
    nginx:
        image: nginx:1.11
        ports:
            - "8080:80"
        volumes:
            - ./:/app
            - ./.docker/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
    db:
        image: mariadb:10
        ports:
            - "33060:3306"
        environment:
            - MYSQL_ROOT_PASSWORD=root
            - MYSQL_DATABASE=app

Configure the application

The application will be running from the container so it may use internal hostnames. For example, to configure a database connection, open your .env file and change the DATABASE_URL to following:

DATABASE_URL=mysql://root:root@db:3306/app

Because we don’t run any extra services right now, this is the only change in the application.

Initialize the environment

To initialize the whole environment, use a docker-compose tool.

docker-compose up -d

It may take a while since the docker has to build a custom php image. After a few minutes, everything should be done and the application should be accessible under the localhost:8080 url.

The up command is for scaffolding the whole environment. For the normal usage, you should use start/stop commands.

Final result

If you try to reach the localhost:8080, you should see the page similar to the following.

Symfony 4 default page running on the new environment.

To connect easily to the database server outside the environment, you may use the localhost host and the port 33060. For instance, the below screen presents connection details in a Sequel Pro application.

Database connection details in Sequel Pro application.

Summary

We successfully went through the first part of the environment preparation and configuration. This is the very beginning part of the whole journey. Each environment should be suited for the application it hosts. Maybe you’ll use a different database engine, a different cache storage or a queue solution. The configuration should depend on the needs.

Is it worth to create a universal environment? It depends on how universal it should be. If you want to develop applications in only one framework or one technology, why now. If you would like to create a versatile environment for a bigger spectrum of technologies, it may not make much sense.

That’s the reason why I didn’t write the final configuration with a paragraph of text describing how to use it. Instead, I tried to show you how to set up an environment you really need. I have taken each action as the response for a need.

Preparing the environment is different than the programming, but also makes fun. It’s worth to know what you use it, how, and even why. Stay tuned.

Featured photo by chuttersnap on Unsplash

I've finished studies. Here are some of my thoughts

Mon, 24 Sep 2018 00:00:00 +0000

A long time since my last article. During the last few months, I was fully involved in my last big project at my university. Dozens of articles and books, hundreds of visited pages and thousands of words later I had finished it and a few weeks later I defended my master’s thesis.

Since I took a short break from almost every activity, I had a lot of time for thinking. When I look at 2012, I see the young guy who knows everything he needs and he wants to only the paper to confirm it. Now, in 2018, I’m equipped with knowledge, a completely new point of view and goals in life. But it wasn’t so obvious even during my education.

Some people asked me what studies gives me at all. Was it worth to spend almost every weekend for the last 6 years at the university? Do I feel the difference? There are no simple answers to these questions because they depend on the approach. And it changed over time. But let me sum up.

Knowledge

The main thing that studies gave me was a wider look at the life. What does it mean? Thanks to the studies, I’m more aware what I don’t know yet so I reduced the number of things I don’t know that I don’t know. Moreover, I’m a kind of person who likes to knows how things work and why. Discovering and learning something new gives me a lot of fun.

I have respect for all people who continue Ph.D. and decided to learn and teach others. I thought about this path but without full devotion, it doesn’t make much sense to me. At least not now.

My expectation when it comes to studies were related to the extension of my skills as a professional software developer. What is interesting, the best things I learned weren’t related directly to programming itself. Moreover, knowledge in those others fields gave me a fresh look at some problems in my disciplines. I still like to connect dots.

I decided to study at weekends because I had wanted to start a regular job. I couldn’t sit at home. In the office, I had an opportunity to acquire the knowledge from other people. Personally, I don’t have an impression, that thanks to my studies, I have a good job. The education was important, but my own work, self-education, and researching approach played the bigger role. It doesn’t mean that I disregard the influence of studies on my life. It’s hard to be objective in this field.

Lectures

In spite of the attractiveness of some lectures and exercises, the lecturer itself has a big impact on the quality of the lesson. A lot of things matter: the way of speaking, the presentation, contact with the audience, and the general attitude to own job and the knowledge. Unfortunately, it was rare to find a single person with all of these features but when somebody had most of them, it also conducted the most interesting lecture. I still don’t know if it’s a rule or case.

People

I had pleasure met a lot of fantastic people – both lecturers and other students. They were so supportive and they shared knowledge willingly. Otherwise, I had to deal also with cruel and unfriend ones. All these contacts gave me value and learned how to deal with them.

Too often people pushed an only minimal effort to pass through everything that is mandatory. In the beginning, it was too easy to work this way, but later the requirements were high enough to say about them as challenging.

One of the favorite ways of conducting projects is to assemble people to so-called teams to work on some project together. It often meant that everything was done by one or two people on the team while the rest of the team were doing nothing but the documentation. Universities should push more effort to encourage and teach people how to work as a part of the team.

Improvement of soft skills

However some lessons were dedicated to practice soft skills like public speaking, presenting etc, in the vast majority, it was a waste of the time. A few hours during whole studies are not enough to learn students how to prepare a good talk. If someone didn’t learn somewhere else how to do it, he ended up speaking to the sleepy audience, sometimes including the lecturer.

The reason is, 90% of the time, people spend listening to someone who is also learning as others. There is no way to get knowledge how to do it well except a few comments from the lecturer. The rest of the time you may spend conducting your own talk. But talking to the group of people that don’t want to listen is unpleasant.

People need a friendly environment and the coach. Someone who see the posture and movement. Someone who listens to each word. Not to count the number of mistakes but to show how to improve them. I wish met someone who teaches me how to speak to people well.

Thesis defense

I had a tendency to repeat the same mistakes again, especially when it comes to writing a thesis. I wrote two: Bachelor’s thesis and Master’s thesis. I hadn’t wanted to explore something I already knew. I treated this part of the study as an opportunity to learn something completely fresh for me. Regardless of the topic, I had a constant problem with starting doing something. A procrastination is a good word here. I procrastinated (and I’m still doing it) a lot. The Parkinson’s law is very brutal.

Summary

This post is based on my personal experiences. You may have different opinions and feelings and this is completely fine – if you want, you can share them ~~in comments~~ on twitter.

Every time I hear that somebody talks to other about the sense of computer science studies, I recall why I started them and what I got at the end. And I have no regrets.

Diploma degree or the title don’t really matter. The only thing you’ll use after studies is everything you learned during them. Keep learning. The more you know, the more opportunities you see.

Featured photo by Suad Kamardeen on Unsplash

The code is a common good, so be responsible for him

Sun, 15 Apr 2018 00:00:00 +0000

I joined to the project that was developed by one guy. He was an amazing developer with plenty of ideas and skills. He had also the best knowledge about the system – its domain, architecture, used solutions, hidden tricks, and workarounds. In every single task, I needed his support because I didn’t understand how things worked. My main goal was to retrieve as much knowledge as possible from his head. The reason was simple – his contract is ending in a month. I wish I didn’t know about it before.

The project had the long to-do list of features. Some of them were partially implemented. Some other functionalities had specified time–frame because of the seasonal nature of the project. Everything was important from the business point of view.

There were no tests, no code’s style guide, no documentation (besides a few out of date README.md files across repository). I didn’t want to touch anything because the code was unstable and cause a lot of side–effects. After one month of torment, I thought – “f*ck it, let’s make it works”.

I share this story because it was one of the most important moments in my career as a software developer. The moment when you realize that if you want to go further you can’t hide behind lack of knowledge and misunderstanding of someone’s else solutions. I decided to explore it and treat it like my own code. I provided fixes, improvements, and changes. Finally, I took pleasure in making things better.

That led me to another thought – the code is common good.

Working as the only one developer

I don’t know how popular is working as the only developer in a project. On the beginning of my career, I worked in this way for a very short time. Most of the projects were like single–shots – do it and forget it. It was very convenient because it allowed me to test my ideas. Otherwise, It was hard to care of the code if you had limited amount of time and the pressure from the top.

It was a lie. Almost always these projects came back both for simple improvements and further development. Even if they didn’t come back to us, they were transferred to other companies. The legacy code I produced wasn’t a problem for me – it was a problem for other developers. I realized it when I received the legacy project the first time.

Although working as a one-man band has a lot of advantages, it has also a lot of pitfalls. The one is very destructive – you cannot challenge yourself with other developers. You have limited chance to speak about the project, about the potential solutions, you cannot validate your ideas and nobody does the code review for you. You are behind the wall so you are fully responsible for your work and the education. Your commit messages are the only single letter, you have plenty of jokes in comments in code with the hope that someone will see it and laugh. You don’t write tests because you know that everything works as expected.

If this description fits you, be sure to read on.

Consequences of the bad code

If you work as a professional developer, the code which you produce belongs to the company. Even if you are the only one dev in the project, you work for (or on behalf of) your company. If the project is successful, the team may grow.

Working in the team is a completely different story. But let’s focus on the code.

Every project has an entry barrier – it depends on the complexity of the project and domain but also on the code style, documentation, technical debt, tests, and all used hacks and workarounds. If you don’t take care of these things, the entry level will be higher and higher. The introduction of a new person to the project will take a long time causing extra costs for your company.

New developer in a project. Photo by Lionello DelPiccolo on Unsplash

Also, when a developer is leaving a company, somebody else has to come to his place. Leaving the project without any documentation, tests, with the code in a state resembling a rubbish dump is very selfish and irresponsible. It isn’t only a harm to the company but also for coworkers and the rest of developers.

Our goal as professional software developers is to create software that supports business, resolves problems and makes money. Software which produces other problems isn’t a good investment.

We should take care of the code all the time during our daily work. The most difficult situation is when the team is split to people who don’t take care of the code and the rest, who take the responsibility for what they produce. Whom to blame if something goes wrong? Is it really a team?

Another phenomenon is the disconnection from the code written by someone’s else. Some developers avoid changing this code because they don’t feel responsible for it. Since the code is in the code base, it’s a common good of the whole team. We all can comment and blame but only a few people are able to give a constructive feedback or even help.

Is it all the consequences? I think the topic isn’t exhausted yet. But they are problems that I already met. I’m sure that you can write a few more comments.

How to be a responsible developer?

I wanted to ask “How to write a good code?”, but this is very subjective and depends a lot on technical aspects. Instead, I decided to ask about responsibility, because a responsible developer writes a good code. It’s a natural implication.

I don’t know the easy-to-follow guidelines that make every developer accountable. That’s something that everyone has to discover. Instead, I have prepared a simple list which bases on my experience.

1. Treat code as a common good

No matter if you are only one developer or you are working in the team. The responsible developer always keeps the code in the good condition. Try to reduce the technical debt and make the code more understandable. Don’t disconnect yourself from someone’s else code – treat is as your own and makes it better.

2. Don’t blame your teammates

Because the code is common, in case of error, the whole team fails. You can always check what someone else introduces to the code base. Don’t you have a code review process? It’s a nice time to make it as a part of the development process.

3. Write code easy to read

Developers spend most of the time reading the code. Make this process easier by writing code which is easy to read and understand. How to do this? For instance, I use the return early method to make my code more readable.

4. Write code easy to remove

Isn’t something necessary? Remove it. Code which is easy to remove is also easy to change. Less code in the code base means less code to read and maintain. The good code’ deprecation strategy is very helpful.

5. Write tests

This is a proof that the code works as expected. It’s also a good documentation. Don’t treat tests like separate tasks, it doesn’t make much sense.

6. Deliver solutions, not problems

Even if you don’t have a good information to your manager, product owner or somebody who is in charge, try to show other potential solutions, not only problems.

7. Don’t be a blocker

Don’t you want to learn something to improve your skills? Do you block opportunities for others? Instead, take as much as possible from them. Don’t you take care of the code style? Take advantage of auto-formatters – it isn’t a big deal.

8. Don’t be an asshole

Be kind and helpful. Simply. Even if your contract is ending and you don’t have good relation with your managers, try to do everything to make the work easier for your teammates and future developers. You can meet them in one of the next companies.

Summary

I started from a short story about the situation that had a big impact on my career. The key is to understand that companies don’t hire people to help them, they hire because they need help. They are looking for someone who is able to solve their problems and who introduces something fresh to the organization.

I could wait and complain that the code is ugly and unmanageable. The company didn’t take care of the good knowledge transfer. But I didn’t. Instead, I did my best to make the work better for me and other developers. Ultimately I changed job but I left leaving things in the best form.

Purposely I don’t write about the tools and methods how to write a good code. It isn’t a key in this article. The point is to realize that we all more than just developers – people who write code. We solve complex problems, we give our customers the value, we make someone’s else life easier, we work together – whether we like it or not. Let us respect our work and take responsibility for what we do.

Featured photo by Štefan Štefančík on Unsplash

Did your team define own code deprecation strategy?

Mon, 12 Mar 2018 00:00:00 +0000

I didn’t always work in a larger team. At the beginning of my journey as a computer programmer, I was the only one person in a project. It meant that I had had a free-hand (or semi-free-hand) to choose how I could write a code and which solutions I could use. From day to day I could perform a little revolution in the codebase. No consequences and no problems because the only user of the code was me.

Although this situation might look as the best case for a programmer, it doesn’t. Especially for the junior programmer. You have no opportunity to learn from someone else. You can’t validate your ideas and thoughts with others. Ultimately your only friends and co-workers are Google and Mr. StackOverflow.

After some time, I got a new job and I started to work as a part of the team. As a new person in the company, I had tons of ideas and tools that we could use in a project to make our work better, more pleasant and easier. It was relatively easy to introduce new features into the codebase. On the other hand, it was almost impossible to remove the old ones. Why?

Because we didn’t have the code deprecation strategy.

If you add a revolutionary feature to the codebase and your co-workers like it, you can be 100% sure that further changes will not be that easy. That works quite good, why we want to change it? More people in the project, the more difficult is to introduce ground-breaking changes.

That’s the difference between working alone and with the team. Even if you introduce something that should replace the current feature, people should also accustom to the new one and change the old code as well. If not, your project will contain a deprecated or even obsolete code.

Why code become obsolete?

By saying obsolete, I mean that a code which is not in use anymore. Often it has left in codebase because it was used somewhere, but things changed and people forget about it. The code can become obsolete when we introduce a new way to solve a particular problem but we don’t do anything to get rid of the old feature.

So every time when we:

have completely new idea to solve the problem,
discover that that current implementation would be better,
change the infrastructure behind,
need to change something to adjust the system to business needs

There is a chance to left old code, even if it’s not needed anymore. Before code becomes obsolete, it enters the transition phase – the stage when we change the old implementation to the new one. Problems might appear when we allow the old code work or exist in parallel with the new code without control. Finally, we can forget about the old code. Sometimes we consciously leave old code in the project just in case.

So what we should do to avoid situations like above? We should write code that is easy to understand, easy to use, and easy to replace. We should encourage people to use the new one all the time.

How to change the code?

Let’s take the following statement as true:

It’s very difficult to replace every usage of function/method/class in the codebase but it’s relatively easy to implement the new function/method/class beside old one and switch a usage to the new implementation step by step.

According to above, we shouldn’t change the code directly within old methods or classes. Instead, we should create new methods and classes which will replace older counterparts in the future. If you follow SOLID principles, it’s convenient, thanks to Single Responsibility Principle and Open-Close Principle.

The cause is simple. If we don’t replace all usage of the old function or class and we change something in the old code (implementation, method’s header, etc.), then the execution may fail.

I’m going to show a simple scenario when we perceive an opportunity to improve the usability of the code. Let’s say that we have a class called ArticleRepository.

class ArticleRepository extends BaseRepository
{
   /**
    * @param User $user
    *
    * @return Article[]
    */
    public function getArticlesForUser(User $user): array
    {
        $query = $this->db->prepare("SELECT * FROM articles WHERE user_id = :id");
        $query->bindParam(':id', $user->getId());
        
        $result = $query->execute();
        
        return $this->mapToEntities(Article::class, $result);
    }
}

Method getArticlesForUser is very easy to use – its only need a User object. In our system, we use this object in many places, more often in a magic way (it make refactoring by IDE much harder).

After some time we realized that we have to create whole User object just to pass it to this method only to retrieve its id. However, chaning the implementation or function header may lead to errors if we miss some occurrence. Instead, we can implement the new method and change the old one to work as a proxy.

class ArticleRepository extends BaseRepository
{
   /**
    * @param User $user
    *
    * @return Article[]
    */
    public function getArticlesForUser(User $user): array
    {
        return $this->getArticlesForUserId($user->getId());
    }
    
    /**
     * @param int $userId
     *
     * @return Article[]
     */
    public function getArticlesForUserId(int $userId): array
    {
        $query = $this->db->prepare("SELECT * FROM articles WHERE user_id = :id");
        $query->bindParam(':id', $userId);
        
        $result = $query->execute();
        
        return $this->mapToEntities(Article::class, $result);
    }
}

Great! We can use the new method without break anything in our code! If we had cases where we would like to use the old method, we could leave it. But let’s assume, that the getArticlesForUser method isn’t vital anymore and it should be removed. Shall we do it now?

Deprecate the old method

As we previously mentioned, we couldn’t find each place where the old method is called. In spite of all, leaving the method (even commented – who is care about comments?) in the code is like the invitation to use.

Happily, thanks to the PHPDoc standards, we have a @deprecated annotation. Moreover, the majority of modern IDEs recognize this annotation and mark the corresponding calls to inform and discourage developers to use it.

PHPStorm notifies when you use deprecated methods or class.

The only thing we need is to add a @deprecated annotation to the DocBlock comment above the method. It’s also recommended to add extra information about the future of this method and potential replacements.

class ArticleRepository extends BaseRepository
{
   /**
    * @param User $user
    *
    * @return Article[]
    *
    * @deprecated in favour of {@see getArticlesForUserId}. It'll be remove in the next major release.
    */
    public function getArticlesForUser(User $user): array
    {
        @trigger_error("Method getArticlesForUser is deprecated and it'll be removed in the next major release. Use getArticlesForUserId instead.", E_USER_DEPRECATED);
        
        return $this->getArticlesForUserId($user->getId());
    }
    
    // ...
}

Bracket syntax within the PHPDoc means that the tag within is embedded as inline to refer to other parts of the documentation.

We also put the trigger_error function with preceding silence operator (@) to simply notify users whenever a method is executed. Thanks to this, we can set up a custom error handler dedicated to collect deprecation errors. It’ll give us a lot of useful information.

A custom handler may look like below.

$deprecationHandler = function(int $no, string $message, string $file, int $line, array $context) {
    $trace = debug_backtrace();

    $message = date("Y-m-d H:i:s") . ": {$message} in {$file}:{$line}." . PHP_EOL;
    $message .= "Context: " . json_encode($context) . PHP_EOL;
    $message .= "Trace: " . json_encode($trace) . PHP_EOL . PHP_EOL;

    $file = fopen("deprecation.log", "a");
    fputs($file, $message);
    fclose($file);
};

set_error_handler($deprecationHandler, E_USER_DEPRECATED);

Of course, it’s only an example to show which data we can retrieve. You can use whatever you want – whatever you choose, Monolog should cover all your needs. The most interesting part is a debug_backtrace()function which lets us see where the method call comes from – it’s usually done by third-party handlers like Sentry or Bugsnag.

When removing deprecated code?

By deprecating code we set a contract:

/* @deprecated in favour of {@see getArticlesForUserId}. It'll be remove in the next major release. */

So, please keep your word. If you don’t make it because of problems, deadlines, whatever, please update the obsolete information! There’s nothing more misleading than deprecated comments and deprecated promises about deprecated features.

Does my team need a deprecation strategy?

You may ask: Does my team (or even I) need a code deprecation strategy? “My team is small enough and our project is small yet and evolve dynamically”. It doesn’t depend on the size of a team or even project. It’s about the maturity of every member and the whole team. Moreover, it’s related to how we solve problems.

Our solutions may look like hotfix every time and there’s not a field to use a deprecation strategy. It’s not a way to work at all.

By applying well–tested and checked principles to the work we can produce good quality code and solutions that are open to following improvements. Also, the whole process evolves so we can push next improvements like deprecation strategy to it.

If the code changes dynamically and we have to be sure that everything works as expected then we need an easy and understandable system that helps us remove old features. That’s what a deprecation strategy is about – removing the code in a save way.

Maybe you develop an open–source library whose use thousands of developers. You may have tons of ideas about further development, new features and improvements to existing ones. But if you want to change something you have to give people easy to use replacement and encourage them to change. You can remove the functionality in the next major release causing astonishment in many people. You can also depreciate the functionality and announce the replacement to slowly accustom people to the new solution.

The same situation happens when you work in a team. Developers also used to some solutions. We should encourage them to change but it needs time. That’s why it’s worth to keep the old implementation working. It’s not the only reason. It’s simply hard to perform a global change across the whole system.

I loved the way how Symfony team delivers features that don’t have a backward compatibility. They don’t change interfaces, that could break everything. They manage by using a simple deprecation strategy. We can learn from them.

Summary

All processes and principles should make our work easier. The deprecation strategy is like another contract between developers in the team – I’m going to remove it, but I have a better alternative for you. We can also deliver new functionalities and improvements without worrying about the previous features, however, we will have to maintain one thing more.

Needless to say, that keeping the codebase at the minimum is very important. Less code to read and understand make our work easier. An obsolete code is worse than legacy because the legacy code at least works and we often know when we break something.

Don’t hesitate to remove old code, if it’s really old and needless. But don’t forget about the Principle of least astonishment. If something is in use but it needs to be replaced by the other, give people the alternative and depreciate the former feature. Try to encourage and accustom them to the new implementation. If it’s really better, people will catch up quickly.

And how often do you remove old code? If you had joined to new team and project, where everything is completely new to you, would you like to explore the obsolete code? Probably it’ll be confusing. Therefore, maybe is worth to remove code that we don’t have in the codebase?

Resources

Don’t use properties and methods from outside the interface

Mon, 26 Feb 2018 00:00:00 +0000

It may sound obvious or even weird for people who programming in statically typed or compiled languages. In my work, I use PHP which is a dynamically typed language with optional strict typing introduced in version 7. At the beginning of my journey with PHP, I didn’t care so much about typing. I had a trivial cause — they didn’t exist yet.

I used to write a code without thinking about types. It was convenient and fast. Furthermore, it allowed writing proxy functions which recognize parameters type and it runs the proper function. Yes, overriding doesn’t exist in PHP.

But things changed when I discovered polymorphism. The magic keyword Interface that allows us to define the mandatory set of features in the object. The picture of writing a code without worrying about missing methods was incredible. I still didn’t understand one thing yet. How interfaces help me since I can pass everything on the function call?

This question lied a long time in my head until I discovered I can define the required type of passed object. Since that time, I changed completely the way how I write the code.

My first interfaces weren’t a piece of art. They were ugly, big, and sometimes overloaded by methods. However, they changed completely my approach to building applications. With time and practice, I started creating better, more consistent and useful interfaces.

Unfortunately, PHP gave me a possibility to do nasty things. Something, what is completely unacceptable in statically typed languages.

Let me show you an example.

Imagine we have the ReadableInterface with could be anything where from we can read: stream, file, HTTP resource, whatever we want. Let’s say that we use one single implementation — ReadableFile and we need to store parsed data in the file with a recognized name. Method getFileName exists but only in this specific implementation.

public function handle(ReadableInterface $readable)
{
    /** @var ReadableFile $readable **/
    $fileName = $readable->getFileName();
    
    $parsed = $this->parse($readable);
    
    file_put_contents(__DIR__ . "/{$fileName}.parsed");
}

What did I do? I lied. I lied that we can pass to the function something that is readable. Linters are happy, PHP is happy, what could go wrong? Nothing, unless we pass to this method e.g. ReadableStream implementation which has no getFileName.

This long introduction was on purpose showing the problem with polymorphism with PHP. Is technically possible to use methods and properties from outside the interface. It doesn’t mean we should do it. We should never assume that methods or properties which are not defined by interface will exist within every implementation.

I share this story just to give some context to the problem.
I also produced code like above but I stopped when I realized that is wrong. However, I sometimes notice that people confidently use methods from outside of interface just because they can and it’s easy. It’s not a good way.

What is the interface for?

Basically, the interface is:

set of features, that object must have
a contract that object delivers something but we don’t care which specific object

But the best explanation of Interface I found on StackOverflow. Consider this situation:

You are in the middle of a large, empty room, when a zombie suddenly attacks you.

You have no weapon.

Luckily, a fellow living human is standing in the doorway of the room.

“Quick!” you shout at him. “Throw me something I can hit the zombie with!”

You can read the full answer here.

What does the man ask about? About something that implements interface! He didn’t specify what exactly but he defined his needs.

What is the problem in PHP?

There is not a problem with PHP itself. Maybe a little bit. It will not stop you from using the methods or properties outside the interface.

But the problem lies in us, developers. If we always treat interface as a contract, nothing wrong will happen. However, we often know what is behind the interface and it’s a potential field for abuse. Instead of using “something, that… (behavior)” we use concrete implementation, even unconsciously. Because we can.

How to solve it?

Simply, treat the Interface as a contract. Use only methods which are defined by this contract. Don’t think what it exactly is. As long as it implements the Interface that you expect, it’s OK!

If you really want to use something outside the interface, you should revalidate its. Is it still relevant? Maybe it’s not so useful as it should be? Or maybe should you expect a specific implementation rather than Interface?

Summary

Needless to say, that polymorphism is a powerful tool. It is also one of the most important things in Object Oriented Programming paradigm. The described problem also involves abstract classes, not only interfaces.

There is one case where I decide to partially bypass the interface. Take a look at this example:

interface MessageInterface {...}
interface LoggableMessageInterface extends MessageInterface {...}

interface MailerInterface 
{
    public function sendTo(string $email, MessageInterface $message): void;
}

Imagine, that we want to create LoggableMailer which should be compatible with the MailerInterface. This mailer is able to handle both MessageInterface and LoggableMessageInterface but for the second once, it implements the extra behavior. The easiest way is to check whether passed $message is an instance of LoggableMessageInterface. I’m still looking for a better solution.

Generally speaking, we should use only methods and properties that are explicitly defined by the contract.

Resources

Featured photo by Igor Ovsyannykov on Unsplash.

Why I removed my old accounts and why you should too

Mon, 29 Jan 2018 00:00:00 +0000

Whenever people say about New Year’s changes they, mainly think about resolutions, plans and set up goals. But this is also a great time to small retrospective both your personal and digital life.

Let’s talk about the digital once.

Do you realize how many registrations have you performed over the last year? Or over the last 5 years? Who did you give your email, personal data, photos, documents or other files to? Too often, it’s too much to know about all this.

Think about each website, application or service which you used and you didn’t use it for a long long time. Social profiles, cloud storage, dedicated tools and the other software. In each of them, you probably left some data. Maybe it’s not relevant and doesn’t have any value for you, but this information is about you.

What if one day, someone will get access to all of these data you left in the Internet?

Nothing lasts forever

I know, we all left a lot of data about ourselves only by browsing the Internet and our favorites sites. It’s also interesting but let’s leave this topic for another article. Instead, let’s focus on abandoned accounts.

Services on the Internet grow quickly and it’s easy to switch to another one. All you need to do is create an account. What do you do with your old account? If nothing, it’s like me (until now).

Why don’t I want to remove my old accounts?

I had many reasons why I didn’t remove unused accounts. There are a few of them:

I thought I would need it yet
I’ll not be able to use the same e-mail for a new account
I kept some data there and I didn’t have any better place for them
My friends/family still used this service
I couldn’t find an option to close the account
I had a really nice username 😉
Just in case…
Why should I delete my old accounts?

Time passed and I forgot about old accounts. I created new and abandoned old ones. Therefore, I received tons of newsletters, marketing messages, and other stuff from many companies. They knew my name, e-mail address, birthday and maybe another personal information. Some of these probably know much more. And nobody gives me the guarantee that my data is safe.

I thought I have to do a little cleanup, but first, I had to face my doubts. That was not easy, but I figured out a good explanation for each of them.

1. I think I’ll need it yet

Yes, that’s the common problem. I figure out the solution I’ve started following a similar rule which I use with newsletters. If I don’t use a specific account but I think I’ll use it in the future I add it to the quarantine. If I don’t use it during this time, I’ll remove it.

2. I’ll not be able to use the same e-mail account for a new account

How likely is that I’ll come back to this service in the nearest future? If the probability is high I leave an account as is but I remove my data from it.

Information on Twitter site.

If you use Gmail, you can modify your existing email address by adding some dots or a plus sign.

Maybe it’s worth to put this account to the quarantine, the same as in the 1st point.

3. I keep some data there and I don’t have any better place for them

I’ve rejected this excuse quickly. My computer, backups or cloud storage on an account which I use regularly. Abandoned accounts aren’t a good place for my data.

4. My friends/family still use this service

Do I lose a contact with my friends or family if I delete my account? Probably not. Moreover, don’t I use this account anymore, yes? So, I probably already have an alternative way of communication. There is no excuse.

5. I couldn’t find an option to close the account

Some old services don’t have an option to close or remove the account. The only solution is to write an email to the help desk with an account removing requests.

eople are looking for how to delete a specific account. Shouldn’t be this option easier to find?

If you have a problem with finding an option to remove an account, you can check the site called Justdelete.me. It is “a directory of direct links to delete your account from web services” with most popular services and useful information how to perform the account deletion process.

6. I have a really nice username

It wasn’t the most important factor in my case, but maybe in your case will it be?

If you are proud you have an account with a cool username, such as your first name, a single letter and so on, you can keep it. That’s your choice. Many services don’t allow reusing before used usernames. But if you don’t use an account anymore is worthwhile to keep it just for keeping the username?

7. Just in case

Sometimes I can’t find a proper reason why I should keep an account. Maybe this is a reason why should I delete it?

8. Why should I delete my old accounts?

Each service may contain a lot of data about their users. It depends on what it exactly is. Since simple services don’t know too much, bigger ones like cloud storages, social media and e-mail accounts possess more data.

Let’s do a quick think exercise.

Do you want to store a private information within abandoned accounts?

What if someday somebody gains access to your old account? Will you regret it? Will it hurt you in some way? Will you be ashamed of data that will leak, e.g. photos, notes or archival chats? If you’re in doubt, you’re on a good way.

Let’s summarize. You don’t need it. Nobody else shouldn’t need it.

What did I find in abandoned accounts?

Some of my old accounts, e.g. Avast or Ubuntu One contain information about the system and my other devices. Moreover, Avast kept more data like my last known IP address or even space on my hard disk.

My avast account had some data about my old computer.

On my old AirDroid account, I found the information about previously connected devices.

My old devices linked to my unused AirDroid account.

Although they contain neither personal nor important data, I just simply remove these accounts. That’s the idea behind the clean-up. Besides that I found a couple of accounts which contains more interesting data:

My old e-mail account, with dozens of messages and information about other accounts and services,
Wunderlist.com account with my old lists and tasks,
Pastebin.com account with plenty of notes containing archival chats, logs, code snippets and so on,
StridesApp account with some old entries,
RescueTime account containing information about my computer, software usage, visited pages, etc.
Bookmarks at Chrome with tons of URLs which I’ve never visited anymore.

Above list is not completed yet, but should give you a small preview, what kind of data you could have left somewhere on the Internet.

Clean-up time

If you decided to perform some cleaning – good choice! I’ll give you some hints at the beginning.

Start by identifying your old accounts. Accounts that you used to use but you completely forgot about them. Social media, online games, forums, instant messengers, online tools and services and so on.
- You can also review the list of connected apps to your Google/Facebook or the other OpenID account.
- If you use a password manager, you can run through entries to see which of you can delete.
- Check your e-mail. If you don’t clean your inbox regularly, you’ll be able to find a lot of useful information about lost or abandoned accounts.
Sign in to each of this account, download all data which is relevant to you.
Remove an account. If you are in doubt, you’ll come back to my list to see how I deal with it. Or you can write down your own reasons why you don’t want to delete old accounts.

Summary

Since we don’t have any special reason why we should remove old accounts, most of the people don’t care about them. But data breaches are real and we have to be aware of them. The fewer places where our data are, the lower chance of the potential leak is.

Sometimes it’s hard to get rid of all the old accounts because not every service gives a possibility to remove the account. But after 25 May 2018, when GDPR (General Data Protection Regulation) becomes effective, we as users and customers will give a powerful right called right to erasure. Time will show, whether it’ll be effective or not.

I encourage you to take the responsibility of your digital life in your own hands. You can follow this simple rule: “If you don’t need it, remove it.”. We’re living in the digitalized world where every piece of information matters. We shouldn’t disregard it.

Resources

Featured Photo by Markus Clemens on Unsplash.

Inbox zero – efficient way to manage emails

Mon, 08 Jan 2018 00:00:00 +0000

Every day at least 30 new messages land in my mailbox. Newsletters, reminders, status updates, mentions, recommendations and even personal or business messages – each of them need my time and an attention. And no one besides me is responsible for handling them.

I had my very first email account on the public news site. I received tons of advertising messages until I got the invitation to the Gmail. This mail service was capable of identifying unwanted messages. That was what I have been looking for for a long time.

Although I receive a bunch of messages each day (more important than spam or advertisements), at the end of the day my inbox is empty. Thanks to Inbox Zero approach, I manage my matters without wasting time. It helps me preventing myself from being overwhelming by a number of emails.

I didn’t have any special system for managing my inbox. I deleted less important messages. Sometimes I left them in inbox without the reason or if messages have looked non-trivial. That way I had all messages marked as reading but the counter still shows more than 1000 messages in the inbox. It seemed to me, that these messages would be useful in the future. The fear that I’ll need something I before removed kept me from deleting. I needed something more efficient.

What is inbox zero?

Inbox zero is one of this system. The system that helps you keep your inbox under control. It doesn’t take a lot of time, it only requires a lot of discipline. First, Inbox zero assumed, that email is only a communication channel. Each incoming message should be proceed until the empty inbox.

Instead of interrupting by every message that arrives in your inbox, you should check for new emails and process them in a bulk. The list of operations you can perform on a single letter is as below:

DELETE
DELEGATE
RESPOND
DEFER
DO

As you can see the list is short but it contains all actions that are you need to keep your matters under control. I’m going to explain each one later in the article.

Advantage of Inbox zero approach

Treating email only as communication channel decreases the number of places where you can look for the next task to do. Although keeping information in the inbox may look convenient, it may lead to negative results, e.g. procrastination. Checking your inbox without special reason and seeing the same messages, again and again, force your subconscious mind to work on them. Even if it’s not necessary at this moment. Each day you see a lot of information that requires processing, so why you want to add another one?

The second advantage is feeling, that you have everything under control. Imagine, that you look to your inbox and the counter shows 0 messages. Give yourself one minute and try to do this exercise in your head. How do you feel? Relaxed, easy, calm? Or maybe anxious, scared, afraid? First time can be strange but it’s normal, especially when you are not used to deleting or reading each email. Not processed emails are an invitation to losing time:

“I don’t completely read this newsletter”…
“Nice photo, I remember that”…
“I forgot to answer”…
“I read this funny story once again”…

Don’t let yourself waste a time. Take a look from the other side. The empty list of messages should tell you that you probably run a wrong program. Or maybe you check the e-mail too often? If it’s nothing to do within your inbox you’ll go to the next task.

Another advantage is searching. The more messages you delete, the easier potential seeking is. It doesn’t matter how advanced search engines we have. Sometimes there will show you a message that you don’t need but it can attract your attention (and time).

I always leave my inbox empty. That makes me more relaxed and happier. I know all things to do I’ll find in other, more reliable place, designed for planning. Moreover, I don’t miss something important. For me email is for communication, not to organize life.

How to work with Inbox zero?

1. Clear your inbox

Inbox by Google – Do you recognize this screen?

If you want to try Inbox zero approach you have to clear your inbox. You have to realize. It’s inbox ZERO. Nothing can be in your inbox. You have to realize that you don’t need all these messages. It’s not a cloud drive. Your photos, documents, and other important stuff can stay in the dedicated cloud drive, like Dropbox, Google Drive. I have tips for you – there are 3 different ways how to reach empty inbox:

Hardcore version – select all your messages and delete them. Fastest and easiest approach but I don’t know if somebody went this way.
Select messages older than 3 months and delete them. You have to revise the rest of emails and process them. You can adjust the time to yourself, but don’t be so sensitive. If you don’t need these messages for 3 months, probably you’ll never need there in the future.
Hardcore version – process all your messages.

The choice is yours. It depends on how much time you’re going to spend sorting your messages and also how important information lay in your inbox.

2. Don’t sit in your inbox

It’s nothing interesting to do in your inbox. Don’t waste the time keeping the mail application open waiting for the next message. Instead, keep your email app closed and check it for new emails from time to time. Email is not a real-time communication. You don’t have to respond immediately (unless your work or life requires that).

3. Turn off email notifications

Notifications are everywhere. Sometimes are helpful but most of them are useless. Notification is a guy, who interrupts you when you are in the middle of the ground-breaking idea. Nobody wants to be distracted in this moment. You probably either so don’t let interrupt yourself.

Each time you got a notification, you switch the context – and context switching is the productivity killer. Moreover, switching to process only one single email is a waste of time. Avoid it. Multitasking works in the car, but it’s no efficient during the work.

4. Unsubscribe newsletters that you don’t even read

Do you ever know how many newsletters you subscribe? How many of them do you read? Or maybe you open them for marking as reading? If you got newsletters that you know you don’t read – unsubscribe them. If there is no option for that, mark it as a spam. When you have a doubt, you’ll do this simple trick: check how many times in a row you don’t read the newsletter. If it’s more than 3, you’ll unsubscribe it.

5. Don’t leave a message without the action

Now, I’m going to say more about actions you can perform on a single message, but instead of focusing on each operation I’ll focus on the flow instead.

If the email doesn’t need any action – DELETE or ARCHIVE it.
If the action will take less than 2 minutes:
1. simply RESPOND. Try to write short emails.
2. simply DO. That’s probably not a big deal. 2 minutes of your time is nothing compared to the time that your mind spend to thinking about it.
If the action will take more than 2 minutes:
1. DELEGATE, if this task can be done by someone else.
2. DEFER, and I recommend place other than the inbox. Extract the most important part of the message, redefine it in your own way and put it into reliable place:
  1. Task list, if it’s a project or bigger task to do.
  2. Calendar, if it’s an appointment or something it’s scheduled.

This flow let you take a decision for each message you receive.

6. Don’t leave your inbox without processing all messages

Once you notice something in your inbox, you open a new “loop” that resides in your mind. Although you don’t think about it, your subconscious mind process it taking resources.

What is loop?
“Loop” comes from "Getting Things Done" philosophy David Alen. It means a matter that has not been completed and still sits in your head, even if you don’t think about it. Sometimes it comes back for a while in random moments and it can break your way of thinking.

Try to close as many loops as you can even if it means that you have to DEFER something for later (outside the inbox). It’s also a good decision, better than nothing. Do you want to come back to the same message again and again? If you have doubts just ask yourself – Is any reason why I can’t process this message?

7. Safe people’s time as much as yours

People don’t like long emails. Nobody has time to read them. Because of that, you shouldn’t write a novel – you probably also ignore this kind of emails. Be safe in words and try to write the message that you want to receive (when it comes to the volume, not the content). Don’t beat around the bush.

It’s not a piece of advice about the email netiquette. I assume, that you already know how proper email should look. Short messages are easier to digest and process. Keeping it simple gives you a chance, that your letter doesn’t fall down to the bottom of the inbox.

Summary

Although I don’t process hundreds of e-mails every day, I feel that I work in the most efficient way with my inbox. After introducing this system into my life I spend less time in the mail app. I also stop worrying about things I didn’t complete. Paradoxically, I check my inbox less frequent and I spend much less time than before. I’m able to process more emails and quickly getting things done.

This system is not for all kinds of job. If you have to have continuous contact with your customers, you work at help desk or support or your job is strongly connected to e-mail – I’m sorry, it happens. But if you are lucky and your job doesn’t need yours all the time sitting in your mailbox, that’s great. If you have read this text and it interests you, maybe it’s a good reason to give yourself a chance to check it in practice. Perhaps you’ll discover other benefits (or defect). I would like to hear what do you think about it and how it works for you.

I have to also remind, that I’m not a coach or a personal trainer. I’m sharing only things that I tested on myself and that work for me. Please note, that everything I have written here dapted to my lifestyle. I can’t promise, that all them will work for you but I’ll keep my fingers crossed. So, do you already know when you’ll be cleaning your inbox?

Resources

Featured photo by Xavier Massa on Unsplash.

Building the associative array – ideas

Mon, 18 Dec 2017 00:00:00 +0000

When we talk about arrays we usually mean the vector of something – primitives, objects or even arrays. But there are a lot of situation when we need to carry extra information with our data. We can use nested arrays but it doesn’t cause that the specific item within the array will be easy to identify. To achieve it we should use custom keys. They provide easy access to any element of the array as long as we know the key corresponding with that value.

There are some reasons why we can use associative arrays instead of simple vectors, e.g. readability. Named keys are also more meaningful than numeric indexes. Instead of thinking about the whys, image the situation when we could use a hashmap. Let’s focus on how we can build the associative array from a vector.

Example scenario

In our application, we have users from different parts of the world. During the registration process, the user can select its own country. In our database, we store information about the country in the ISO-3361-alpha-1 format. The user should see the full country name instead.

We actually have an array of Country objects, which definition is presented below:

class Country
{
    private $name;
    private $code;

    public function __construct(string $name, string $code)
    {
        $this->name = $name;   
        $this->code = $code;
    }
    
    public function getName(): string
    {
        return $this->name;
    }
    
    public function getCode(): string
    {
        return $this->code;
    }
}

One of our frontend components can render Combo (knew as the <select> node) and it expects associative array as an input in format ['value' => 'text visible for user'].

We need to create this array.

Approach #0: the basic loop

function prepareCountriesForCombo(array $countries): array
{
    $output = [];

    foreach ($countries as $country) {
        $output[$country->getCode()] = $country->getName();
    }
    
    return $output;
}

$countriesSelect = prepareCountriesForCombo($countries);

The most basic approach I know. I often question solutions which use the raw loops because they are low-level being. But if they are encapsulated within the well-described function, that’s O.K.

Creating a function designed to produce specific data for other component seems to be a nice idea. Let’s consider that we need to deliver data to different Combo. It would be nice to use the same function but with different data. I need to generalize this approach.

Approach #1: Generic way to produce associative array

Besides the array containing the data, we need to pass two extra functions. The former will be responsible for transforming the element to the text visible for a user and the latter will transform the element to the key of the associative map.

The function could look like this:

function mapForCombo(array $array, Closure $fnValue, Closure $fnKey): array
{
    $output = [];

    foreach ($array as $element) {
        $output[$fnKey($element)] = $fnValue($element);
    }
    
    return $output;
}

$countriesSelect = mapForCombo(
	$countries, 
  	function (Country $country) { return $country->getCode(); }, 
  	function (Country $country) { return $country->getName(); }
);

It looks pretty good and probably we should stop discussing more this boring feature. After all, let’s try to find other implementations for this specific case.

Approach #2: use array map functions

I really like array functions. They are so powerful. We can transform keys and values of the associative array independently and we can combine them into the single array.

function mapForCombo(array $array, Closure $fnValue, Closure $fnKey): array
{
    $keys = array_map($fnKey, $array);
    $values = array_map($fnValue, $array);

    return array_combine($keys, $values);
}

$countriesSelect = mapForCombo(
    $countries, 
    function (Country $country) { return $country->getCode(); }, 
    function (Country $country) { return $country->getName(); }
);

The code is much shorter, but under the hood, PHP has more work. Although there is no any visible loop in the userland, PHP engine must iterate over the array at least 3 times (twice for map and once for combine). It seems slow and not so efficient.

Approach 3: array map by reduce

The next approach may be to use an array_reduce function instead fo mapping. What this function exactly doing I describe in one of my previous posts. Simply, this approach is to reducing array to the single value. But nobody said that the single value couldn’t be an array!

function mapForCombo(array $array, Closure $fnValue, Closure $fnKey): array
{
    return array_reduce($array, function ($acc, $element) use ($fnKey, $fnValue) {
        $acc[$fnKey($element)] = $fnValue($element);
        return $acc;
    }, []);
}

$countriesSelect = mapForCombo(
    $countries, 
    function (Country $country) { return $country->getCode(); }, 
    function (Country $country) { return $country->getName(); }
);

The only thing we should do is to create a function which will assign the transformed value to specific key in the array (similar as on 1st approach). The loop logic is hidden within the array_reduce function so only care about the correct assignment.

Which solution is better?

Tests

Personally, I don’t like do and read benchmarks for things that usually doesn’t matter. Otherwise, I collect some ideas and I want to validate them in such way. That’s why I test which implementation work faster.

Test file and environment

I prepared a simple test file, you can view it on this GitHub Gist. Although it’s not so complicated, it’s enough to see which implementation is better, relatively to each other. In short – I execute each function 5000 times and I measure the duration of the execution.

First I tried to run this script on my local environment, but I came to the conclusion, that the bettser option is to use a completely different server. I rented a droplet 2GB RAM / 2 CPU from DigitalOcean which ready-to-use LAMP Stack on board and I run the test.

root@lamp-2gb-nyc3-01:~# php -v
PHP 7.0.22-0ubuntu0.16.04.1 (cli) ( NTS )

The test generates an output which I copied and paste as a CSV to the Excel. It allows me to easily calculate the average time and plot a simple chart.

Test results

I have to admit, that results were really surprising to me. First, please take a look at the final outcomes.

Approach #1 – foreach – 0,283s
Approach #2 – array_map – 0,253s
Approach #3 – array_reduce – 1,054s

Differences are more visible on a simple bar chart.

The chart presenting average duration of each approach.

My first impression was something like a mixture of confusion with misunderstanding. The solution that seemed to be the best turned out, that it was the least efficient. It shows, that we should always test and validate our idea.

I personally use the array_reduce approach quite often. Fortunately, the test was deliberately exaggerated to shows differences between each approach. For smaller datasets, differences are not noticeable.

It doesn’t mean, that you should not use the array_reduce function. It has specific use cases and meaning so don’t hesitate to use this function if it fills your needs.

Summary

I’m not sure, whether this experiment will change the way how I write the code. To be honest, it’s only a micro-optimisation on a bigger scale. Although sometimes it does matter, e.g. if you have to write high-efficient algorithms, for most of us it’s just an unnecessary concern.

Otherwise, it reveals something completely different. We believe that things work faster because it seems to be a faster. They execute fewer loops and they contain much less complicated logic and conditions. This example shows that there is no truth in all cases.

As you notice, there isn’t a golden rule to how to transform data to the associative array. Simplicity is the key, but we always should test, confirm and compare our solutions with other. It’s not only about the array transformation but also about many other things. Maybe your next solution will be better, easier and faster?

Resources

Featured photo by G. Crescoli on Unsplash

What is idempotence?

Mon, 04 Dec 2017 00:00:00 +0000

If you frown after reading idempotence, don’t worry. Although this word sounds unusual it conceals a simple mathematics property. Needless to say, this characteristic has also a big meaning in Computer Science.

Idempotence operation is an operation that can be applied multiple times without changing the result beyond the initial application. It may sound difficult but after a couple of examples, the general concept will be obvious.

To be honest, I spent a lot of time looking for proper examples of idempotent functions or operations. Although I caught the idempotence concept very fast, I wanted to go deeper. I tried to understand not only what it is but also why does it matter. I’ll share my thoughts on this article.

Idempotence in mathematics

In math, each function produces a certain output. This is the reason why we ever use functions – we pass input arguments and we expect an output value. If we pass the same argument, again and again, each time we’ll get the same output. It means that math functions are pure – their output depends only on the input arguments.

$$f(x) = y$$

Now, imagine that we call the function with a certain argument. It produces a specific output depends on the input value. Let’s call the function again, but for the input value, use the result from the previous execution.

Idempotent functions

If the output doesn’t change anymore, the function is idempotent. Only the first execution of function produces a new value. Each subsequent call returns the same value, no matter how many times we call the function. In other words, the function fulfills below equation.

$$f(f(x)) == f(x)$$

Do you know the example of an often used idempotent function? The well-knowing example is the absolute value. For each negative value as an input abs returns its positive counterpart. For each positive value, it returns input argument as is.

$$abs(abs(-5)) == abs(-5)$$

Pure function and idempotent function are not the same. Function can be pure, but not idempotent, e.g. f(x) = x + 1. Essentially, f(x) is pure because its output depends only on the input argument. But it’s not idempotent because it not fulfills the condition f(f(x)) == f(x).

Idempotent operations

Under certain conditions, we can recognize some mathematical operations as an idempotent. Take a look at these two binary operations: addition and multiplication. They are not fully idempotent. But if one of the argument is an identity element, the operation is idempotent.

For addition, the identity element is equal to 0 (zero). Whatever adds to zero, we’ll get the same value.

$$5 + 0 = 5$$
For multiplication, the identity element is equal to 1 (one). Something times one always returns that “something”.

$$5 * 1 = 5$$

It doesn’t matter how many times we’ll perform the operation with identity element in the pair. Each time we’ll get the same output.

Idempotence in programming

In programming, we have a lot of pure and idempotent functions in mathematical sense, e.g.

floor / ceil
lowercase / uppercase
trim

In many programming languages, functions do much more operations beside returning an output. Sometimes they return nothing. Another time they return different results, although we use the same input arguments.

But why? What is the difference between the mathematical function and the programming languages functions?

Side effects

Of course, we can place maths functions within our program and everything will be fine. But in most cases, we use functions to manage the state. These functions that depend not only on the input arguments. They also use something outside its local scope (e.g. global state) so they are no longer pure – they have side effects.

The function has side effects when

it reads or modifies the global state
writes or reads data from I/O or database
uses a timestamp, current date
uses something that has effects beyond it

But side effects doesn’t exclude function to be idempotent. We’ll take a closer look at the non-pure but idempotent function.

Idempotent functions in programming

In programming, a function is idempotent, when we can call it many times with the same arguments and the result won’t change beyond initial calling.

It is close enough to the mathematical definition but instead of applying function, we execute it many times.

Let’s assume, we have an array of figures. Now, we’ll sort this array in PHP language. Please notice, that PHP’s sort function expected input array as a reference. That means, it modifies the variable outside its scope, so the sort function has the side effect.

$numbers = [3, 4, 2, 5, 1];

sort($numbers); // [1, 2, 3, 4, 5]
sort($numbers); // [1, 2, 3, 4, 5]
sort($numbers); // [1, 2, 3, 4, 5]

It doesn’t matter how many times we’ll call a sort function. We’ll always get the same outcome. The same for well-knowing setters.

$person = new Person("Simon");

$person->setName("Szymon");
$person->setName("Szymon");
$person->setName("Szymon");

Only the first call of setter will change the name. If we use it many times, even by mistake, nothing bad will happen.

Idempotence in HTTP

Idempotence plays a big role in HTTP protocol. Lots of web browsers lead to the concept of idempotency and the assumption which HTTP methods are idempotent and safe. According to the official HTTP specification, GET and HEAD should be considered as safe methods. Moreover, GET, HEAD, PUT, DELETE, OPTIONS, and TRACE should be considered as idempotent methods.

Thinking in REST

Let’s assume, that we have a resource. When we read from it, nothing dangers should happen. This is exactly how GET method works. It should only read from the server and that’s why it’s safe.

Please note, that we talk about the official HTTP/1.1 specification. Everyone could implement these methods differently, according to own needs.

Now, let’s consider a PUT method. This is idempotent operation. Only the first request should affect our resource. If we’ll send an extra request with the same data, nothing destructive should happen, because data is replaced, rather than appended.

DELETE is also interesting. The first DELETE request is obvious – it deletes the resource from the server. But each next DELETE request to the same resource should be ignored. The server should respond to the client by the specific code, e.g. 404. That makes this request idempotent.

But not all HTTP methods work in this way. Let’s take a look at the POST method. This method is neither idempotent nor safe. It creates the new entity within the resource, identifiable by a unique id. Each subsequent POST request will be creating the new entity with completely new identity, so this method isn’t idempotent.

Browser notifies if you try to resend the form.

It’s so important that even web browsers show an alert in the case when we try to resend before processed form. Thanks to the HTTP/1.1 – RFC 2616, each browser handles all methods in the same way.

Conclusions

Idempotence is not a magic tool that we should use to build better things. It’s a property of some functions and operations that protect us from errors caused by repetition. Very often we don’t even need this characteristic within your own functions. That’s O.K. Idempotence plays a big role not only in math and computer science but also in the real world and it often went unnoticed.

Did you notice the featured image? What will happen if we press the same button in an elevator again? In elevators I know, nothing will happen. The floor will be still highlighted and the elevator does its work.

Buttons at pedestrian crossing work in the same way. It doesn’t matter how long you keep it pressed or how many times you push it. The very first call (in each cycle, of course) is important.

Photo by PICSELI on Unsplash

If you select the current channel on your remote, the TV will do nothing.

Some cars have a remote with two buttons dedicated to opening and closing a car. If you open the opened car, it’ll be still opened. That’s obvious.

Do you know other examples? As you see, they are everywhere but I thought long before I found some good ones. Idempotence isn’t so important in our daily work but is worth-knowing property if you use the functional paradigm. Now you know why.

Resources

Idempotence – Wikipedia
RFC 2616 – Hypertext Transfer Protocol – HTTP/1.1
Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software, Side-effect-free functions, (Polish edition)

Why should you return early?

Mon, 20 Nov 2017 00:00:00 +0000

When I started my journey as a computer programmer, I had written my code in various way. In most cases, I translated requirements into specific conditions and statements, but I hadn’t taken too much care of readability or quality of my code. “It works, that’s it” – and the code base grew and grew in time. Thousands of written methods later, I had discovered that a lot of these little pieces of logic could look better if I reverse some condition within them. Unconsciously, I’ve started following the rule which I had later called the “fail fast” rule, and subsequently “return early”.

I realized I hadn’t had the one preferred way to check conditions within the single method. Sometimes I let execution if the expression was true. Another time I had waited for the false result to show the error message before the main part of the code was executing. In my head, I discovered various styles of checking conditions.

Update 2020-03-14

Although this article is 2yo, it’s one of the most popular articles I’ve written on my blog. I decided to review it, correct mistakes and simplify the content by eliminating visible inaccuracies. The topic this article touches is timeless, so I hope you’ll enjoy the reading. I appreciate every feedback.

Two approaches to check requirements

The most basic style of checking if something meets the requirement is to put the expectation into the condition simply. There is one example. I want to send a message only if the variable $email is the valid e-mail address and if the variable $message is not an empty string. Otherwise, I want to throw an exception.

function sendEmail(string $email, string $message) 
{  
    if (filter_var($email, FILTER_VALIDATE_EMAIL)) {
        if ($message !== '') {
            return Mailer::send($email, $message);
        } else {
            throw new InvalidArgumentException('Cannot send an empty message.');
        }
    } else {
        throw new InvalidArgumentException('Email is not valid.');
    } 
}

The above snippet is one-to-one what I wrote before. Is it correct? Definitely. Is it readable? Not quite.

What is wrong with moving requirements literally to the code?

We can point two problems with sendMail method:

It has more than one level of indentation within the method body.
You don’t know the success path without analyzing the flow of the method.

Since the first point concerns the “Clean Code” side, let me focus on the second one. The code mentioned above has a couple of branches and the return statement isn’t visible – you have to look for it. I know that this example is short, however, try to imagine the more extended method with more arguments and conditions (or even probably more than one return statement). Have you tried? Some pieces of code are awful, but the small change could improve readability drastically. The only thing you have to do is to reverse conditions.

function sendEmail(string $email, string $message) 
{  
    if (!filter_var($email, FILTER_VALIDATE_EMAIL)) {
        throw new InvalidArgumentException('Email is not valid.');
    }

    if ($message === '') {
        throw new InvalidArgumentException('Cannot send an empty message.');
    }

    return Mailer::send($email, $message);  
}

I admit that I also remove one else after the second condition. I know that it’s subjective, but in my opinion, this code looks much cleaner and prettier than the previous version. And the only thing I did is reversing conditions within if statements.

What I reach by reversing conditions?

First, the max indentation level within method body is one, and we got rid of the nested condition. Thanks to these operations, the code is more readable, and it’s easier to understand. The expected result is at the end of the method, so it’s easy to find. The flow within this method is clear and visible.

Now you probably guess what I meant by calling this approach “fail fast”. Unfortunately, I figured out that Jim Shore, together with Martin Fowler, had defined “fail fast” before I created my very first “Hello World” app. The goal is almost the same, but I decided to change the name of my approach to “return early”. I also figured out that this is already defined pattern. However, I’m glad I had noticed it on my own! Because of this, I’ll continue using the other name, which, in my opinion, is more accurate for this article.

Now, let’s get a closer look at the “return early” concept.

“Return early” concept

You should know what is the “return early” rule. Nonetheless, I want to try to define this concept by myself

Return early is the way of writing functions or methods so that the expected positive result is returned at the end of the function and the rest of the code terminates the execution when conditions are not met.

I wrote the first version of the definition in 2017, but it was awkward and complicated, so I decided to make it simple. The old one sounds that:

“Conception of writing functions (methods) so that the expected positive result is returned at the end of the function and the rest of the code should cause as soon as possible the termination in case of divergence with the purpose of this function.”

If you see a way to improve this definition, please let me know. I’m always open for discussion.

Follow the “happy path”

“The pot of gold is at the end of the rainbow”. Do you remember this story? Maybe did you find one? The expected positive result should be like a pot of gold – placed at the end of the function. Just look at the above sendEmail example. Our goal was to send a message to the given e-mail address. The execution must follow the “success path” or, as the header says, the “happy path”. Happy, because this is the expected positive result and this is the reason why we call this function.

The main path within the function is clearly visible.

By placing the final operation at the end of the function, we always know what our “happy path” looks like and what is the expected positive result. No guessing, no analyzing – quick look should be enough to find out what is the goal.

Get rid of bad cases early

The “happy path” is the best result that we got from the function. But the execution might don’t go in the right way – and that’s ok. You can come back to our sendEmail example. If someone provides an invalid e-mail address, then he’ll cause an expected bad case. We don’t want to pass an invalid e-mail address to the further execution. We should terminate the execution or simply return from function earlier.

The termination could be either the return or the exception. The choice is yours. You should use the way that is more convenient for you or what it would fit in a particular case.

Many people may feel awkward by using more than one return statement per single function. If the language gives you a possibility to explicitly define return value, I don’t see any reason why you shouldn’t use it, even more gladly when it gives you real advantage, e.g. better readability and maintainability.

The Bouncer Pattern

Bouncer Pattern is what I described in the previous section. It’s also known as “Assertions” or even “Guard clauses”. All mean the same and they prevent code execution in case of the invalid state. Take a look at this code snippet:

function matrixAdd(array $mA, array $mB) 
{
    if (!isMatrix($mA)) {
        throw new InvalidArgumentException("First argument is not a valid matrix.");
    }

    if (!isMatrix($mB)) {
        throw new InvalidArgumentException("Second argument is not a valid matrix.");
    }

    if (!hasSameSize($mA, $mB)) {
        throw new InvalidArgumentException("Arrays have not equal size.");
    }

    return array_map(function ($cA, $cB) {
        return array_map(function ($vA, $vB) {
            return $vA + $vB;
        }, $cA, $cB);
    }, $mA, $mB);
}

The first part of the code is responsible for checking if an operation can proceed. The return statement contains the main logic of our function. In the ideal world, you wouldn’t need the first part of the code.

You may noticed that I use array functions instead of basic loops. I recommend this approach in another article.

Return early from the controller’s action

Controller’s action is a great candidate to use a “return early” approach. Actions often do much more things before they return a final response. Let’s consider below updatePostAction method that comes from the PostController.

/* PostController.php */

public function updatePostAction(Request $request, $postId)
{
    $error = false;
    
    if ($this->isGranded('POST_EDIT')) {
        $post = $this->repository->get($postId);
    
        if ($post) {
            $form = $this->createPostUpdateForm();
            $form->handleRequest($post, $request);
    
            if ($form->isValid()) {
                $this->manager->persist($post);
                $this->manager->flush();
    
                $message = "Post has been updated.";
            } else {
                $message = "Post validation error.";
                $error = true;
            }
        } else {
            $message = "Post doesn't exist.";
            $error = true;
        }
    } else {
        $message = "Insufficient permissions.";
        $error = true;
    }
    
    $this->addFlash($message);
    
    if ($error) {
        $response = new Response("post.update", ['id' => $postId]);
    } else {
        $response = new RedirectResponse("post.index");
    }
    
    return $response;
}

Such a big piece of code here. It has one return statement and a lot of nested conditions. I don’t know what you feel looking at this code snippet, but I think that it should be a better way to do this. Another example takes advantage of the “return early” approach.

/* PostController.php */

public function updatePostAction(Request $request, $postId)
{    
    $failResponse = new Response("post.update", ['id' => $postId]);

    if (!$this->isGranded('POST_EDIT')) {
        $this->addFlash("Insufficient permissions.");
        return $failResponse;
    }

    $post = $this->repository->get($postId);

    if (!$post) {
        $this->addFlash("Post doesn't exist.");
        return $failResponse;
    }

    $form = $this->createPostUpdateForm();
    $form->handleRequest($post, $request);
    
    if (!$form->isValid()) {
        $this->addFlash("Post validation error.");
        return $failResponse;
    }
      
    $this->manager->persist($post);
    $this->manager->flush();
    
    return new RedirectResponse("post.index");
}

Now you can observe that this action has a clearly visible “happy path”. It’s not one-to-one as the previous examples, but it follows the same conception. If you expect that something may go wrong, check it and return early! Moreover, the one-level of indentation makes the code more readable, and, just note, I don’t need to use the else keyword anywhere!

Return early from recursive functions

Also recursive function always should have a terminator that can looks like “return early” statement. This is a simple example:

function reverse($string, $acc = '')
{
    if (!$string) {
        return $acc;
    }

    return reverse(substr($string, 1), $string[0] . $acc);
}

Objections about “return early” conception

The main question is – what is the advantage of this concept? Why should I take care of this? Or when I shouldn’t?

Problem 1: Code style is subjective

We, as programmers, spend much more time reading code than writing. That’s common true. Therefore, we should write code which is easy to read and understand. I proposed to react on falsy state first, not only because it improves readability but also it ensures that the main gear within the function is always at its end (or really close). In other words, we see the expected success result as our “happy path”.

Indeed, the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code. …[Therefore,] making it easy to read makes it easier to write.

– Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship

But the code style is subjective. Someone may feel better with single return statement or with multiple assignments instead of writing the same method call again and again (like addFlash in one of above code snippets).

Could you imagine these two hypothetical functions? Which of them seems to be simpler?

Problem 2: Sometimes “return early” is an exaggeration

I know a lot of cases where applying “return early” approach could harm the code or at least gives nothing. Think about all these setters where you have to assign value to the field only when the param is different than false:

public function setUrl($url) 
{
    if (!$url) {
        return;
    }  

    $this->url = $url;
}

That’s not the best way to improve code’s readability. You should know where and when you can use techniques like " return early “. This situation shows that sometimes it’s good to ignore rules.

public function setUrl($url)
{
    if ($url) {
        $this->url = $url;
    }
}

Problem 3: It looks like break statement

Because function may have more than one return statement, we don’t accurately know where from the result comes. As long as you use “return early” to stop reaching an invalid state, you shouldn’t care about this. Of course, are you writing tests?

Problem 4: Sometimes it’s ok to use a variable instead of many returns

There are some structures where “return early” doesn’t change anything. Please look at these two examples of the same function. It comes from the “game of life” algorithm where function calculates the next state of the cell:

function nextState($currentState, $neighbours)
{
    $nextState = 0;

    if ($currentState === 0 && $neighbours === 3) {
        $nextState = 1;
    } elseif ($currentState === 1 && $neighbours >= 2 && $neighbours <= 3) {
        $nextState = 1;
    }
    
    return $nextState;
}

The same function but using “return early” looks as below:

function nextState($currentState, $neighbours)
{
    if ($currentState === 0 && $neighbours === 3) {
        return 1;
    }

    if ($currentState === 1 && $neighbours >= 2 && $neighbours <= 3) {
        return 1;
    }
    
    return 0;
}

So, as you see, there is no big advantage in this case.

Are you wondering what is a “happy path” in the above code snippet? That’s the exciting thing. The function’s name isn’t so meaningful, but its hidden goal is to “kill” a cell. Everything that keeps our cell alive goes to the invalid state. So, maybe do you have a better proposal for the name for this function?

Conclusions

The way how we write code comes from our customs. It’s hard to say that something is better than others when the outcome is the same. It’s very subjective like many more things in the programming.

“Return early” concept introduces some rules. Rules that you shouldn’t blindly follow. As you can see there are a lot of cases where this approach helps and otherwise, where it doesn’t change anything. It’s crucial to keep a discipline during writing code because the consistent code style is much more important than anything else when it comes to the cognitive load.

I use “return early” as often as I can. It means that I use it where I see the advantage of returning early from functions, e.g. because of invalid state or when the change improves readability. I’m used that success is always at the end of the function. The hidden mantra sounds:

Follow the success path (happy path) and react in case of errors.

Resources

Featured Photo by Franz Spitaler on Unsplash

My start with React and its rich ecosystem

Mon, 06 Nov 2017 00:00:00 +0000

I remember how I was excited discovering how much frontend frameworks like AngularJS could make my work easier. Even though I was a strong backend-oriented programmer I decided to learn the concept of Angular. I dug into it at ease and I started using this tool in my daily work.

Since I changed my job I lost the opportunity to use frontend technologies regularly. However, I’m still fascinated about this part of software development and I follow trends and latest changes. Therefore AngularJS 1.x was officially abandoned I started looking for something new and fresh.

I didn’t look for a long time. When I first saw React library, I fall in love with it. I quickly caught the concept behind this tool, thanks to ubiquitous tutorials and examples. I read a lot about the flux pattern, redux, components but I didn’t chance to use them in something more ambitious than <HelloWorld/> component. Till now.

Project conception

Because I still study and I have some projects to do, I decided to smuggle some new stuff this way. This is a really nice opportunity to try something new because of at least two reasons:

You have a specified goal of project.
Your project must work.

Simple, but enough. These rules prevent the further work from being abandonment in the half of project.

My project is a simple online password manager with client-side encryption. This sounds like a good playground for a React application. It also needs some server-side work but first I want to focus on the front-end.

I started working with React

When I first tried to work with React I was frightened how much work it needs to set up a simple environment. Some time ago I tried it once again using an official tool create-react-app. A fresh React project was ready for several seconds. That’s impressive.

Bootstrap comes in

Because of time limitation, I was looking for some existing component libraries dedicated to the UI development. I didn’t want to reinvent the wheel. After a small research, I choose the reactstrap library which wraps Bootstrap components into React ones. Great.

Reactstrap logo.

I also started developing my custom components. This is one example:

class PasswordField extends React.Component {
  constructor() {
    super();
    this.state = {hidden: true};
  }

  toggleVisibility() {
    this.setState((prevState) => ({hidden: !prevState.hidden}));
  }
  
  render() {
    return (
      <div className="PasswordField">
        <InputGroup>
          <InputGroupAddon>
            <FaEye className="PasswordField-Icon" onClick={this.toggleVisibility.bind(this)}/>
          </InputGroupAddon>
          <Input value={this.state.hidden ? '∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗' : this.props.password} readOnly={true}/>
        </InputGroup>
      </div>
    );
  }
}

Needs for routing

After prototyping one screen I realized that could be nice to have a router to add some interactions to my application. Another research led me to the react-router library. Some examples from documentation gave me a good playground.

React Router logo.

I’m a little bit scary reading how these things change over time. According to official documentation, react-router have been completely rewritten. It makes a migration to the newest version difficult.

State management

Subsequently, I added redux to my application. I learned about the concept behind this in past mostly from articles and video courses. Dan Abramov created a brief video introduction to redux – it’s a great starter. Now I check there in the action and I feel that without the good background it would be hard to use it ad hoc.

Redux logo.

In React, I have to describe all changes explicitly. Even when it comes to binding input into the component’s state. Switching from two-way binding to this is awkward but I know problems with bindings within AngularJS applications. After that, I prefer explicit solutions. Redux is intuitive. All possible transitions in the state are described within the reducers. Reducers are activated by actions. Finally, actions are emitted by interactions. The beauty of reactivity.

Conclusions

I was under huge impression how easy I can write a full client application without worrying about the server-side yet. I prepared some actions dedicated to gathering data from the server, but now instead of doing request, I resolved a delayed promise with dummy data. I know all these things were possible but I never tried work in this way before.

The main screen of my application. Very first prototype 🙂

Building with React gives me a lot of fun because it’s interrelated to object-oriented programming as well as to functional programming. It depends on who handles the state management. I also say building instead of programming. I notice a majority of work is a compose bigger components from smaller ones. It may be an illusion because lots of logic is hidden within their as well as within the actions and reducers.

The one thing I miss is tests, because of limitation of time I focus mostly on building the app rather than getting to know the full process. Because of that, I know I have something to catch up in this area.

Featured photo by Braden Collum on Unsplash

Why you should use array functions instead of basic loops

Mon, 23 Oct 2017 00:00:00 +0000

Loop is the basic construction in each programming language. All repeatable operations are being done using loops. I think you know how the basic loops like while, do and for work and when you should use one of them. But are you sure? Maybe should you use array functions?

How many of your operations in a loop rely on building objects based on the value? Or selecting only a few items from the array using a predicate? Maybe you use array values to calculate something? Whatever you do, read why you should replacing basic loops by array operations.

Update 2019-12-11

PHP 7.4 comes with support for short closures aka arrow functions. I enriched this article with examples that use this new functionality.

Working with for loop

Take a look at this example. Looks familiar?

$tags = ["php", "javascript", "programming", "loop"];
$entities = [];

for ($i = 0; $i < count($tags); $i++) {
    $entities[] = new Tag($tags[$i]);
}

Everyone knows the for loop. It has three control statements: setup, condition, and instruction after each step. But what exactly this code means?

Create an empty array called $entities.
Set the counter $i to 0 (zero).
Iterate until the counter is less than a number of items in $tags array and increment counter each time by one.
Inside the loop, create a new instance of Tag. The counter points the name of the tag from $tags array.
Add the new object to the $entities array.

There are a lot of operations which largely obfuscate the main goal. They are irrevelant. Why should you care about the number of elements, starting point, or the iteration process?

“A programming language is low level when its programs require attention to the irrelevant.” — Alan Perlis

Working with foreach loop

You can clean up the previous snippet replacing for in favor of foreach loop.

$tags = ["php", "javascript", "programming", "loop"];
$entities = [];

foreach ($tags as $tag) {
    $entities[] = new Tag($tag);
}

Much better. All unnecessary operations are hidden thanks to the foreach loop. It creates an abstraction over basic iteration process. Thanks to this, you can focus on your main goal instead of repeat the loop control logic again and again.

Foreach loop creates a tiny abstraction over basic iteration process. It allows you to focus on the main goal and gives more readable code in the outcome.

Look how more understandable this code is by reading it step by step:

Create an empty array called $entities.
Iterate over $tags array and make the current element available by $tag variable.
Inside the loop, create a new instance of Tag which $tag as a name.
Add the new object to $entities array.

No details about the process of iteration, incrementation or even number of elements. Great! But both of these pieces of code have hidden side effect. Can you point out where?

Side effect in computer science is a situation when function or expression modifies variables or resources outside its scope.

The problem lies in loops. Both of loops leave extra variable in execution scope – $i and $tag. The variable $tag contains the last element of $tags array. Therefore it’s recommended to unset unnecessary variable directly behind the loop.

$tags = ["php", "javascript", "programming", "loop"];
$entities = [];

foreach ($tags as $tag) {
    $entities[] = new Tag($tag);
}

echo $tag; 

unset($tag);

You can improve this code, even more, by identifying your main goal. What is the main goal in above example? I defined an array of tags and I want to transform them to an array of Tag objects. Each element from array should be mapped to the new one.

Array functions to the rescue

Introduce mapping

I revealed one of the array operations – mapping. This operation allows you to define a function which will be executed over each element in the array. They create a new array contains the mapped entries. In PHP there is a function called array_map.

We’ll change the code a bit to use the array_map function.

For PHP < 7.4:

$tags = ["php", "javascript", "programming", "loop"];

$entities = array_map(function ($tag) {
    return new Tag($tag);
}, $tags);

Since PHP 7.4:

$tags = ["php", "javascript", "programming", "loop"];

$entities = array_map(fn($tag) => new Tag($tag), $tags);

No loop control logic, no side effects. I just defined a function that should be executed on each element. The returned value of supplied function should return the new element.

The Map function takes an argument and returns the new value.

Now, let’s try to explain this code:

Map each element of $tags array to the new one using a supplied callback function. This function defines how to map single element.

That’s it. There is no step 2. The code is easier to read and understand, and communicates proper intentions.

The result of map function. All items were mapped to the new ones.

Some people don’t like the anonymous function. Just remember that array_map expects a callable. You can pass it by string (e.g. when you want to use the internal function), variable, array – the choice is yours.

Meet the filter function

Sometimes you have a need to select only elements from the array that satisfying defined criteria. To accomplish it you could create a loop which conditionally adds elements to the new array.

$stock = [
    ['name' => 'Phone', 'qty' => 0],
    ['name' => 'Notebook', 'qty' => 1],
    ['name' => 'SDD Drive', 'qty' => 0],
    ['name' => 'HDD Drive', 'qty' => 3]
];

$inStock = [];

foreach ($stock as $item) {
    if ($item['qty'] > 0) {
        $inStock[] = $item;
    }
}

unset($item);

You already know that basic loops have a noticeable overhead. Instead, you should use the array_filter function.

For PHP < 7.4:

$stock = [
    ['name' => 'Phone', 'qty' => 0],
    ['name' => 'Notebook', 'qty' => 1],
    ['name' => 'SDD Drive', 'qty' => 0],
    ['name' => 'HDD Drive', 'qty' => 3]
];

$inStock = array_filter($stock, function ($item) {
    return $item['qty'] > 0;
});

Since PHP 7.4:

$stock = [
    ['name' => 'Phone', 'qty' => 0],
    ['name' => 'Notebook', 'qty' => 1],
    ['name' => 'SDD Drive', 'qty' => 0],
    ['name' => 'HDD Drive', 'qty' => 3]
];

$inStock = array_filter($stock, fn($item) => $item['qty'] > 0);

You probably note that the order of arguments in array_filter function is different than e.g. in array_map function. About the PHP inconsistency may write a book but it goes beyond the boundaries of this article.

In this example, array_filter also takes a callable. This function is a predicate. They must return a boolean value which defines whether element must be included in the resulting array or not.

Representation of the filter function. For “green triangles” it returns true, for anything else it returns false.

If you don’t supply the predicate function array_filter returns all items except false. Maybe is this the reason why the order of arguments is different than in other array functions?

The result contains only elements matched the criteria from filter function.

Reduce array to single value

Maybe you remember a few situation when you need to iterate over an array to e.g. sum any value. Take a look at below example. The goal is to get total points for the team.

$team = [
    ['name' => 'Marry', 'points' => 11],
    ['name' => 'Jacob', 'points' => 7],
    ['name' => 'Joanna', 'points' => 6],
    ['name' => 'Tony', 'points' => 8]
];

$total = 0;

foreach ($team as $person) {
    $total += $person['points'];
}

unset($person);

We’ll switch immediately to the array_reduce function:

For PHP < 7.4:

$team = [
    ['name' => 'Marry', 'points' => 11],
    ['name' => 'Jacob', 'points' => 7],
    ['name' => 'Joanna', 'points' => 6],
    ['name' => 'Tony', 'points' => 8]
];

$total = array_reduce($team, function ($carry, $person) {
   return $carry + $person['points'];
}, 0);

Since PHP 7.4:

$team = [
    ['name' => 'Marry', 'points' => 11],
    ['name' => 'Jacob', 'points' => 7],
    ['name' => 'Joanna', 'points' => 6],
    ['name' => 'Tony', 'points' => 8]
];

$total = array_reduce($team, fn($carry, $person) => $carry + $person['points'], 0);

The first impression – there is a more code than in the previous snippet!

You’re right. But the number of lines/words/characters shouldn’t be the most important criteria. Especially if you want to write clean, understandable and side-effects free code.

Sometimes is necessary to write more words to get a more understandable and side-effect free code.

I hope you have seen a difference in the callback function. There are two arguments. The former, accumulator (I call it $carry) contains the reduced value and the latter keeps the current element. The callback function handles returning the new value of $carry.

Reduce function takes two arguments – element and accumulator. In outcome, it returns a new value of the accumulator.

Value of $carry is also passed to the next callback execution. Finally returned from the array_reduce function after all. The third argument of this function is the initial value of $carry.

The result of reduce function is usually a single value. In this case array of triangles was reduced to the sum of its elements.

Mixed operations during iteration

Let’s consider case when you want to do something with elements which are satisfying defined criteria, e.g.

$players = [
    ['name' => 'Mike', 'level' => 5],
    ['name' => 'Ray', 'level' => 6],
    ['name' => 'Ana', 'level' => 5],
    ['name' => 'Joe', 'level' => 2]
];

$advancedPlayersNames = [];

foreach ($players as $player) {
    if ($player['level'] > 4) {
        $advancedPlayersNames[] = "Adv." . $player['name'];
    }
}

unset($player);

Inside this loop, you could find both filter and mapping. We want to get names of players whose level is more than 4. To accomplish it using the array methods you should split this problem into small parts.

For PHP < 7.4:

$players = [
    ['name' => 'Mike', 'level' => 5],
    ['name' => 'Ray', 'level' => 6],
    ['name' => 'Ana', 'level' => 5],
    ['name' => 'Joe', 'level' => 2]
];

$advancedPlayers = array_filter($players, function ($player) {
    return $player['level'] > 4;
});

$advancedPlayerNames = array_map(function ($player) {
    return "Adv" . $player['name'];
}, $advancedPlayers);

Since PHP 7.4:

$players = [
    ['name' => 'Mike', 'level' => 5],
    ['name' => 'Ray', 'level' => 6],
    ['name' => 'Ana', 'level' => 5],
    ['name' => 'Joe', 'level' => 2]
];

$advancedPlayers = array_filter($players, fn($player) => $player['level'] > 4);

$advancedPlayerNames = array_map(fn($player) => "Adv" . $player['name'], $advancedPlayers);

So as you see there is no big deal. Instead of defining an algorithm to extract final data, think more about the process. It contains steps like mapping, filtering and reducing. Is this code excessive? Maybe. Is it easier to read and understand? I think yes.

Summary

If you don’t use the array functions yet, just try. I highly encourage you to write your loops in this way. You’ll be surprised how easy to understand will be your code particularly for your coworkers or even you after a long long time. This approach brings us closer to the next interesting things – Collections. Personally, I prefer array functions over basic loops. In my opinion, they’re more meaningful and expressive, because the code is more declarative – it means that I define what I want to reach instead of describing the way how to do it.

Of course, if you work on the critical system you should consider this advice. Array functions are a little bit slower than basic loops but I think that there is no significant impact on performance.

Resources

Featured photo by Photo Design Lab on Unsplash.

Synchronize local files with Cryptomator Vault

Mon, 09 Oct 2017 00:00:00 +0000

Among thousands of files on my computer, I have some with special meanings for me. Photos, videos, papers, and spreadsheets – they occupy dedicated places within my filesystem. That’s the problem – they exist only on my hard drive. However, I sometimes need a quick access to these files from my smartphone or other computers. The solution seems to be simple: migrating to the cloud. But what if I definitely don’t want to keep these files outside?

Personally, I don’t tap into the cloud storage too much. That’s my private choice that may change in the future. But cloud storage is so powerful, cheap and widely available. It would be nice to use available space on free storage providers, especially in a case when I need some data outside the home.

This is the time when encryption comes in.

Meet Cryptomator

Last time I found a nice project called Cryptomator. It’s a client-side encryption software which supports transparent encryption. That means you have easy access to your encrypted files as if they are decrypted. But it doesn’t mean that they are insecure – they are decrypted on-the-fly on demand. Moreover, Cryptomator stores each file separately. That’s cool, cloud-friendly solution!

I started playing with this and I decided to make a small experiment. I’ve tried to write a simple mirroring solution that once a day synchronizes a bunch of my local files to the encrypted Cryptomator’s vault.

Cryptomator nad macOS’ Automator

I use macOS, so I started by running Automator, Apple’s solution to easily create workflows. Automator is the great tool to create a custom task which interacts with the macOS.

Because Cryptomator is a GUI application, I tried to set up a semi-automatic workflow, similar to following

Run Cryptomator
Wait for X seconds and let user manually unlock vault
Copy files to vault
Close Cryptomator

Single blocks within the Automator allow me to create a workflow like above. Unfortunately, it has several problems.

It was not fully automated
Files always be copied regardless of changes
Workflow didn’t recognize the WebDAV destination after the re-run

These problems enforce me to find other solution. Ideally when Cryptomator would offer a CLI access to manipulate the vault directly by e.g. shell script. After a small research, I found the project cryptomator/cli on GitHub. That was exactly what I was looking for.

Synchronize local files with encrypted vault

CLI for Cryptomator allows us to easily start the WebDAV server as an access layer to the encrypted vault. This server could be running in the background. To get a quick access to the files you need to connect to it. I decided to mount it as an external drive. Therefore, I can interact with this virtual space like with a normal directory.

Platform and Software requirements

Java – Cryptomator is written in Java languages
JCE unlimited strength policy files
Cryptomator CLI JAR file

Install JCE unlimited strength policy files

If you use homebrew you can install the necessary policy files by running:

brew cask install jce-unlimited-strength-policy

Unfortunately, it didn’t work for me so I had to go a manually way. According to Readme file, I extracted files directly to the /lib/security. The `` directory depends on your installation but in most cases, it should be in /Library/Java/JavaVirtualMachines/< jdk_version_of_your_pc >/Contents/Home/jre, according to this StackOverflow answer.

Flow of mirroring files to the encrypted vault

The flow is quite simple:

Run cryptomator-cli and start WebDAV server
Create a dedicated directory as mount point within the filesystem
Attach WebDAV server to new directory
Synchronize files from specific directory using rsync
Unmount the WebDAV server
Remove unnecessary directory
Exit Cryptomator

As you see there is no a big deal and basic shell script can handle it. And that was the goal.

Prototype the mirror script

Configuration

First, I set up some configuration variables. They are widely used in the whole script.

CRYPTOMATOR_PATH="/usr/local/bin/cryptomator-cli.jar"
DIR_TO_SYNC_PATH="$HOME/Private"
MOUNT_PATH="$HOME/VAULT_$RANDOM"
VAULT_PATH="$HOME/PATH/TO/ENCRYPTED/VAULT"
VAULT_PASSWORD="`security find-generic-password -a KEYCHAIN_ENTRY_VAULT_PASSPHRASE -w`"
VAULT_NAME="vault"
BIND_HOST="localhost"
BIND_PORT="8198"

I think they are self-described and they don’t need more comments.

Note about variables
Some of above variables are used as constants. In the shell there is no difference between them. As you can see, names of variables describe the value rather than containing the value. Because of that, they are change-resistant as long as the context is matched. For instance, instead of DOCUMENTS_DIR I use DIR_TO_SYNC_PATH, because I could change the directory and the previus name would not match. You can read more about it in my post: Do your constants make sense?.

The one variable is interesting – VAULT_PASSWORD. Instead of hard-coding plaintext password within the script, I use security tool. This is a macOS tool to retrieve data and password from your keychain. In this case, the script is looking for entry KEYCHAIN_ENTRY_VAULT_PASSPHRASE (you can also adjust this name).

Keychain is very restrictive. You cannot retrieve anything from it, even using the security tool. Before the first use, macOS will ask for permissions to keychain access.

Run Cryptomator CLI in the background

Using above configuration you can run WebDAV server which serves the content of your vault:

/usr/bin/java -jar $PATH_TO_CRYPTOMATOR \
--vault $VAULT_NAME=$VAULT_PATH  \
--password $VAULT_NAME=$VAULT_PASSWORD \
--bind $BIND_HOST \
--port $BIND_PORT &

PID=($!)

The PID will be needed in the very last step to finally go server down.

Mount WebDAV server to the userspace

To ensure easy access to the vault you can mount it like a normal storage. The mount point may be anywhere, but I recommend keeping it in your home directory. You will not a root access to mount it. Moreover, the vault contains your private data, right?

mkdir -p $MOUNT_PATH 
/sbin/mount_webdav http://$BIND_HOST:$BIND_PORT/$VAULT_NAME/ $MOUNT_PATH

Tip:
Instead of using mount_webdav you can also use a mount -t davfscommand.

Synchronization private data with the vault

The easy approach is to copy all data from a private dir to the mounted vault. Although it looks fine, it enforces full cloud synchronization, because all files are modifying. Instead of copy, you can use the rsync tool. Below command uses recursive synchronization and preserves modification times, so it’s cloud-friendly. Additionally, you can exclude some files, e.g. temporary files.

/usr/bin/rsync -rvtP --update --exclude "~*" $DIR_TO_SYNC_PATH/ $MOUNT_PATH/

Cleanup

The very last point is to unmount WebDAV server, remove the mounting point and eventually kill the Cryptomator. Let’s do this!

/sbin/umount $MOUNT_PATH
rm -rf $MOUNT_PATH
kill $PID

Put these parts together

Final script has some extra echo statements for debugging purposes. It has also a couple of timeouts and checks to ensure that every step has proceeded properly.

I noticed that sometimes WebDAV is not mounted correctly to the file system. I don’t know the reason, but as a workaround, I created a loop that tries mount the vault after the short pause.

n=0
until [ $n -ge 5 ]
do
    echo "Try $n: Mount http://$BIND_HOST:$BIND_PORT/$VAULT_NAME/ to $MOUNT_PATH"
    /sbin/mount_webdav http://$BIND_HOST:$BIND_PORT/$VAULT_NAME/ $MOUNT_PATH && break
    n=$[$n+1]
    sleep 1
done

If the vault is mounted then we’ll exit the loop, thanks to the logical ANDoperator and break statement. As you see, the maximum count of trials is 5, so in the worst case, the script will continue the execution without a mounted vault. To avoid synchronize files with non-mounted directory I created an extra check.

if /sbin/mount | grep $MOUNT_PATH > /dev/null; then
    /usr/bin/rsync -hrvtP --update --exclude "~*" $DIR_TO_SYNC_PATH/ $MOUNT_PATH/
    /sbin/umount $MOUNT_PATH
else
    echo "WebDAV not mounted correctly..."
fi

Eventually, the whole script looks like below.

#!/bin/sh

CRYPTOMATOR_PATH="/usr/local/bin/cryptomator-cli.jar"
DIR_TO_SYNC_PATH="$HOME/Private"
MOUNT_PATH="$HOME/VAULT_$RANDOM"
VAULT_PATH="$HOME/PATH/TO/ENCRYPTED/VAULT"
VAULT_PASSWORD="`security find-generic-password -a KEYCHAIN_ENTRY_VAULT_PASSPHRASE -w`"
VAULT_NAME="vault"
BIND_HOST="localhost"
BIND_PORT="8198"

/usr/bin/java -jar $PATH_TO_CRYPTOMATOR \
--vault $VAULT_NAME=$VAULT_PATH  \
--password $VAULT_NAME=$VAULT_PASSWORD \
--bind $BIND_HOST \
--port $BIND_PORT &

PID=($!)

echo "Cryptomator started under PID: $PID"
sleep 5
echo "Create new mounting point: $MOUNT_PATH"
mkdir -p $MOUNT_PATH
echo "Attach WebDAV interface to mount point"

n=0
until [ $n -ge 5 ]
do
    echo "Try $n: Mount http://$BIND_HOST:$BIND_PORT/$VAULT_NAME/ to $MOUNT_PATH"
    /sbin/mount_webdav http://$BIND_HOST:$BIND_PORT/$VAULT_NAME/ $MOUNT_PATH && break
    n=$[$n+1]
    sleep 1
done

if /sbin/mount | grep $MOUNT_PATH > /dev/null; then
    /usr/bin/rsync -hrvtP --update --exclude "~*" $DIR_TO_SYNC_PATH/ $MOUNT_PATH/
    /sbin/umount $MOUNT_PATH
else
    echo "WebDAV not mounted correctly..."
fi

echo "Removing mounting point"
rm -rf $MOUNT_PATH
echo "Close cryptomator"
kill $PID

Link the script to macOS

Store vault passphrase in the keychain

You should store a passphrase to your vault in the keychain. I use a Systemkeychain. After running above script for the first time you’ll be asked for the permission to access this specific entry. To avoid prompting every time when a script is executing, consider adding security to the allowed application list.

Photo by Pixabay

Schedule execution

I’m very used to cron, but I read that this is not a recommended solution in macOS. To define scheduled jobs you should use launchd daemon.

Start by creating the plist job file. In my case this is pl.skrajewski.backup.plist.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>pl.skrajewski.backup</string>

<key>Program</key>
<string>/usr/local/bin/custom/backup.sh</string>

<key>StartCalendarInterval</key>
<dict>
    <key>Hour</key>
    <integer>22</integer>
    <key>Minute</key>
    <integer>45</integer>
</dict>

<key>StandardErrorPath</key>
<string>/var/log/vault-backup.error.log</string>

<key>StandardOutPath</key>
<string>/var/log/vault-backup.log</string>
</dict>
</plist>

This file seems to be more complicated than simple crontab entry but is more powerful. In truth, this is a basic XML file. If you want to know more about the possible configuration options, please take a look on this launchd site.

I briefly describe above file:

Label is a mandatory key and should be unique within the launchdinstance.
Program contains complete path to the executable.
StartCalendarInterval allows you to define the specific time when the job run.
StandardErrorPath is a path to file for stderr.
StandardOutPath is a path to file dedicated for stdout.

Load the job into the launchd daemon

Load the plist definition into the launchd is very simple:

sudo launchctl load pl.skrajewski.backup.plist

And that’s it. The job is currently managed by launchd. If you have an error Path had bad ownership/permissions don’t worry! Change the owner of this file to root:wheel by running the below command

sudo chown root:wheel pl.skrajewski.backup.plist

and try to load the file again. If you want to change the definition of your job, remember about the unload it first using launchctl unload.

Summary

You can find both of this files on GitHub Gists.

Cryptomator is a good piece of software. Moreover, it’s open source. I like the idea of a transparent encryption because it significantly facilitates access to encrypted files. The client-side encryption also put a limit on ways how your data is used by cloud storage providers. This is a double-edged weapon – your data is secure, but nobody helps you if you lost the master key.

When it comes to the proposed mirroring solution – this is only my experiment. I treat this more like curio rather than a way to synchronize files. That’s why I mirror my files instead of storing them directly in cryptomator’s vault. When I wrote this article, Cryptomator CLI was in 0.3.1-Pre-release version. It’s not stable yet but I continue my tests.

I can’t wait for the JavaScript library for the Cryptomator. I’d like to see web-based clients and possible integration with popular cloud storage providers. Client-side encryption is good for folks, not companies that live from their users’ data. Maybe this is the reason why we don’t have this kind of encryption yet?

Resources

Arguments in arrow function explained

Mon, 25 Sep 2017 00:00:00 +0000

ECMAScript 6 has an amazing feature called Arrow Function. It allows writing a function expression faster and smarter than traditional way (using functionkeyword). Worth to know, it’s not a one to one replacement for standard functions.

Everyone knows that the arrow function doesn’t have own this. More precisely: arrow function doesn’t have the own lexical scope. It uses an immediately enclosing scope instead. What does it mean?

var sumMany = function () {
    return [].reduce.call(arguments, function (a, b) {
      return a + b;
    }, 0);
};

console.log(sumMany(1,2,3));

Please notice, that this function use arguments variable, which is the reference to the array of arguments provided to lexical scope of the sumMany function. Let’s try something similar in ES6.

const sumMany = () => [].reduce.call(arguments, (a, b) => a + b, 0);

console.log(sumMany(1,2,3));

Unfortunately, it doesn’t work. We used an arguments variable as before but in this case, this variable is bound to enclosing scope. In this case, it doesn’t have defined arguments. According to the ECMAScript 2015 Language Specification:

Any reference to arguments, super, this, or new.target within an ArrowFunction must resolve to a binding in a lexically enclosing environment. Typically this will be the Function Environment of an immediately enclosing function.

And this is ok! Because arrow function wasn’t designed to be a substitute of a traditional function expression. Probably you have a dozen of places in your code where its behavior was desirable. Just think about all these .bind(this)in your code. Looks familiar?

function UserService() {
    this.findAll = function () {
        return Promise.resolve({data: [{ name: "Jake" }]});
    };
}

function UserController(UserService) {
    this.users = [];

    UserService.findAll().then(function (response) {
        this.users = response.data;
    });

    this.printNames = function () {
        this.users.forEach(function (user) {
            console.log(user.name);
        });
    };
}

There is a mistake in above code, take a look at the usage of findAll function. The nested function has different scope, so instead of assign users to the controller, it assigns theirs to the global object, e.g. Window. To fix it, we should bind a nested function to the enclosing scope.

function UserController(UserService) {
    this.users = [];

    UserService.findAll().then((function (response) {
        this.users = response.data;
    }).bind(this));
}

Or pass reference to correct this, e.g.

function UserController(UserService) {
    var self = this;
    this.users = [];

    UserService.findAll().then(function (response) {
        self.users = response.data;
    });
}

Both of them look hacky and tricky. If you use an ES6, you’ll have an easy way to handle it. By using an arrow function you’ll make your code more understandable.

function UserController(UserService) {
    this.users = [];

    UserService.findAll().then(response => this.users = response.data);

    this.printNames = () => {
      this.users.forEach(user => console.log(user.name));
    };
}

Wherever you need to provide a simple callback function, you may apply arrow functions. They are also very helpful when you want to write the code in a functional style. Ultimately, let’s fix our sumMany implementation

const sumMany = function () {
  return [].reduce.call(arguments, (a, b) => a + b, 0);
};

console.log(sumMany(1,2,3));

Keep in mind when you have to use a function literal instead of an arrow function. It’s very helpful and saves a lot of time.

Resources

http://www.ecma-international.org/ecma-262/6.0/

Blackboxing JavaScript using Chrome DevTools

Mon, 11 Sep 2017 00:00:00 +0000

Lots of JavaScript projects use a dozen of various frameworks, libraries, and third-party scripts. More often code which you produce largely depends from vendors’ work. The good example is jQuery and Lodash libraries which are involved in our pretty code. They are designed to facilitate the work and they are the remedy for especially commonly used, repeatable pieces of code.

This kind of libraries (and frameworks) provide an another abstraction layer behind our code making the execution process more complex. It’s invisible until you’ll need to debug your code because of errors.

Problem

Let’s consider this simple snippet:

let $tableBody = $('.js-table-body');

_.chain(_.times(25, _.random.bind(1, 100)))
    .filter(x => x % 2 == 0)
    .map(x => $('<td>').html(x))
    .map($td => $('<tr>').append($td))
    .forEach($tr => $tr.appendTo($tableBody))
    .value();

This part of code takes advantage of jQuery and Lodash libraries and simply put rows with the random even number into the table. Unfortunately, this logic is mixed together with third party functions. If you put the breakpoint trying to debug the flow of the chain step by step, you’ll be flipped to the library code.

Solution

To avoid exploring the third party code during debugging you can use the feature from Chrome DevTools called Blackboxing. When you reach the file you have to omit, call the context menu on source code and use the option Blackbox Script.

In Call Stack, you’ll have an information that some frames are hidden because of blackboxing the script. Now if you use Step into next function call the debugger will skip all unnecessary frames.

If you want to see the list of blackboxed scripts, you’ll simply go to Chrome DevTools -> Settings -> Blackboxing. You can also add the new blackboxed script by hand.

Tip: Use regular expressions in pattern to blackbox common scripts

As long as you don’t need to dig into some libraries or frameworks you can put them into the blackbox. Prepare your list and put into the pattern list. You can also omit all minified files by using this regular expression

/\.+\.min\.js$

Summary

This is one of the many features in Chrome DevTools that I haven’t used before because I haven’t known it even exists. It only shows that you should get to know better with tools which you use in your work. They are to make your work easy and fast.

Try explain the JavaScript’s truth inconsistency

Mon, 28 Aug 2017 00:00:00 +0000

Everyone has an own version of a truth. I have the own truth, you have your own. JavaScript has many truths and it’s probably no problem until you’ll start to use it implicitly. This article shows and explains one of the popular JavaScript’s trap – inconsistent evaluation in If Statement and Equality Comparison.

Problem

We could start by running simple snippet:

if ([]) {
    console.log("Condition true, check again and is", [] == true)
}

Have you tried? Probably not just once. It’s not a joke from ECMAScript developers, just mechanisms which are used to resolving final value from Expression. In this article, I’ll explain this confusing behavior.

Evaluate an empty array inside If Statement

Take a look at official ECMAScript Language Specification at the 13.6.7 section – The If Statement – Runtime Semantics: Evaluation. A whole document contains a clear description of how all things works in ECMAScript. Right now, let’s focus on this specific behavior.

Our If Statement contains empty array

if ([]) { }

We would consider first two steps from the official specification. The document says:

Let exprRef be the result of evaluating Expression.
Let exprValue be ToBoolean(GetValue(exprRef)).

Our exprRef should be a result of evaluating Expression. However, we don’t need to evaluate anything, because we supplied an object (empty array) in place of Expression.

Subsequently, we need to obtain exprValue which is a result of the composition of two functions GetValue and ToBoolean. I don’t want to dig into the first, just assume, that the result of GetValue(exprRef) is simply an empty array.

Under the hood, the GetValue function resolve path to the value based on the reference. In this example given Expression has concrete Type (not Reference) and there is no need to resolve it.

The last thing before we got the exprValue is called a ToBoolean function with empty array passed through. This function works as a simple switch statement – It returns a boolean value based on matching argument.

Above image presents the ToBoolean results based on the supplied argument. Have you noticed, that there is no Array type in this table? It is nothing wrong with it because what we know as an Array, in JavaScript is an Object.

[] instanceof Object;   
typeof [];

Based on above table, the expected result for Object is True, so that is our exprValue.

if ([]) { }

Equality Comparison with empty array on one side

The next confusing element in this example is the console.log statement containing equality comparison

console.log([] == true);  
console.log([] == false);

It looks logically, but how these expressions were evaluated? The algorithm is described in section 7.2.6 Abstract Equality Comparison in ECMAScript Specification:

ReturnIfAbrupt(x).
ReturnIfAbrupt(y).
If Type(x) is the same as Type(y), then, return the result of performing Strict Equality Comparison x === y.
If x is null and y is undefined, return true.
If x is undefined and y is null, return true.
If Type(x) is Number and Type(y) is String, return the result of the comparison x == ToNumber(y).
If Type(x) is String and Type(y) is Number, return the result of the comparison ToNumber(x) == y.
If Type(x) is Boolean, return the result of the comparison ToNumber(x) == y.
If Type(y) is Boolean, return the result of the comparison x == ToNumber(y).
If Type(x) is either String, Number, or Symbol and Type(y) is Object, then return the result of the comparison x == ToPrimitive(y).
If Type(x) is Object and Type(x) is either String, Number, or Symbol, then return the result of the comparison ToPrimitive(x) == y.
Return false.

In our expression, we have boolean and object value. First matched condition is no 9 which converts a boolean value to Number. According to ToNumberfunction, for boolean it returns 1 when the argument is true, otherwise +0. Returned value is used in place of the old one, so actually, the expression looks like

console.log([] == 1);
console.log([] == 0);

ToPrimitive evaluation

Now, the conditions match to the second marked rule. It’s not possible to quickly compare both of these values because there is a comparison between an object and a primitive value. Therefore we should convert this object to the primitive using ToPrimitive function.

ToPrimitive algorithm for object parameter looks very complex. The second parameter of this function is PreferredType, which is used to set a correct hintvariable. Unfortunately, there is an optional argument and it’s no provided in this function call. Instead, the algorithm use the “default” value, it performs some checks, and later on, it assigns “number” as a hint and passed it together with the object to the OrdinaryToPrimitive function. Its algorithm looks very interesting.

Assert: Type(O) is Object
Assert: Type(hint) is String and its value is either "string" or "number".
If hint is "string", then
1. Let methodNames be «"toString", "valueOf"».
Else,
1. Let methodNames be «"valueOf", "toString"».
For each name in methodNames in List order, do
1. Let method be Get(O, name).
2. ReturnIfAbrupt(method).
3. If IsCallable(method) is true, then
  1. Let result be Call(method, O).
  2. ReturnIfAbrupt(result).
  3. If Type(result) is not Object, return result.
Throw a TypeError exception.

First, we have to construct a methodNames list – for “number” it would contain ["valueOf", "toString"] methods. To get the final value we should execute these methods in order until the result will be a primitive value. Function valueOf for array returns an array itself, so the toString must be called. Finally, our conditions will be executed similarly to these

console.log([].valueOf().toString() == 1); 
console.log([].valueOf().toString() == 0);

Or simpler

console.log("" == 1);
console.log("" == 0);

Right, it’s not finished yet but it looks more familiar and more logically. According to the Abstract Equality Comparison Algorithm:

If Type(x) is String and Type(y) is Number, return the result of the comparison ToNumber(x) == y.

A full description of converting the string to Number is in section ToNumber Applied to the String Type. In this case, an empty string is converted to the 0 value. That’s it.

console.log(0 == 1);
console.log(0 == 0);

Conclusions

The problem with an inconsistent truth in JavaScript results from the process of conversion values. It’s not a simple casting from one type to another, rather a complex selection algorithm working behind the scene. So maybe next time when you’ll be in trouble, try to understand how specific things work. The 5 minutes spent on the investigation will be a better investment than another couple of nasty adjectives that do not give anything.

And remember this simple trick:

if ([] === true) {
    console.log("Try to guess what I mean, or...")
} else {
    console.log("Say precisely what you expect :)");
}

Resources

Do your constants make sense?

Mon, 14 Aug 2017 00:00:00 +0000

Do you use constants in your code? Probably a lot. Even if you don’t define your own is highly probable that you use some constants defined by other libraries or a language itself. Constants are very powerful unless you use it in the wrong way.

Why constants?

Let’s consider some math constants

const PI = 3.14159265;
const E = 2.71828183;

The value of Pi (or e) will never change in the future so it is a great candidate to put it into the constant. How does it look in our programs?

Everyone tells the difference between variable and const. It’s easy to imagine and the names of this two things leave no doubt. The value of constant doesn’t be changed during normal program execution but could be set by the developer, therefore quite often they are used to store configuration data, named states and many other things.

const API_KEY = "4c4712a4141d261ec0ca8f9037950685";
const VERSION = "9.4.1"

class Order
{
  const STATE_NEW = 0;
  const STATE_COMPLETE = 1;

  private $state = self::STATE_NEW;

  public function finish()
  {
    $this->state = self::STATE_COMPLETE;
  }
}

Constants describe the value, so the code is more readable and clean. If you use constants in this way, then it’s perfect. What if someone wants to get rid of all magic numbers and magic strings from code? Probably something like this:

class Printer extends AbstractPrinter
{
    const FORMAT_A4 = "A4";

    public function print($document, $format)
    {
        if ($format !== self::FORMAT_A4) {
            throw new Exception("Document should be in " . self::FORMAT_A4);
        }

        $this->handlePrint($document);
    }
}

Changing value of wrong-named constant

Now imagine that someone has to change the format in the printer, e.g. to B5. For this example, let’s assume, that it have to be done without creating a new implementation. The easiest way to do this is by changing the value of the constant FORMAT_A4:

const FORMAT_A4 = "B5";

It seems at least strange and may lead to misunderstanding and mistakes. The better name for this value is, for instance, PRINTER_FORMAT or simply FORMAT.

The previous example was a little bit exaggerated but it shows the main problem with naming things. We use constants to prevent doing changes in many places so, in conclusion, the name of the constant should be resistant to change their value. Name the constant according to the purpose of its value, rather than the value itself.

Resources

https://secure.php.net/manual/en/language.constants.php

What I learned by doing my simple personal project

Fri, 21 Jul 2017 00:00:00 +0000

In September, I wanted to try something new, so I started a simple project to check my abilities in UX and front-end development. The idea was simple — create customizable, reusable and highly accessible buttons. I have experience with front-end but I usually use CSS frameworks like Bootstrap or Foundation. But not now. I decided to design the whole page completely from scratch.

This post was originally published on Medium platform. You can see the former version directly in medium.com.

Alfred input window with prepared File Filter to look for notes.

The final project is hosted on Github pages. Click on the link below if you want to explore this project. Check the Playground page and play with many variants of buttons!

The Buttons Project – Project of highly accessible and customizable buttons

Are you a front-end developer? Feel free to review my code on Github!

Before I started, I tried to answer these questions:

Why do I want to do it?
Why buttons and not something else instead?
How can I customize these buttons?
How can I ensure accessibility?
How should I start?

My answers are below…

Why do I want to do it?

For learning purposes. Simple. It’s a kind of test for myself. I can say that something is pretty or ugly but I can’t rate my own work. And it’s a great opportunity to try something new (in my case I throw away LESS in favor of SASS).

Why buttons and not something else instead?

I looked for a simple element which has a lot of variants, states, possibilities and use cases. Buttons are omnipresent and they are used on every website.

How can I customize these buttons?

I wanted to change many properties of my buttons without any extra line in CSS (SCSS in this case) files. I tried to list which parameters should be changeable. For each parameter, I wrote set of special CSS classes to change the final appearance.

This approach is followed by Bootstrap, Semantic UI, Foundation, and many other front-end frameworks. But it has one serious inconvenience — use HTML to describe, how the button looks like. It should be the CSS role. Instead, I have to change HTML if I want to change appearance. There is a big advantage — I can insert good-looking buttons on my site without any extra line of CSS.

How can I ensure accessibility?

That was a good question. I didn’t need this kind of functionality and I didn’t know what the problem is. So first I search something information about color blindness. Later on, I tried to emulate various kinds of color blindness to felt it. I had no idea how much this emulation reflects reality but I started to do something with it.

How should I start?

Usually, I start a project by defining dependencies and directory structure. In this case, I tried to keep everything as minimal as possible. I added just things which I was sure that I will use them. My first HTML element on the site was… button.

A month later I had finished and I had decided to publish my work. There are my conclusions about this “simple” project.

I was the victim of “new project” illusion

The idea for a new project is always great. The perspectives for doing creative and interesting things drive to start working as fast as possible, but when I started, an enthusiasm had disappeared when the first big problem appeared.

I questioned my original idea

Sometimes in my mind, a lot of random thoughts disturb me:

“Is it necessary?”
“Are you sure it is that what you should do now?”
“It doesn’t make any sense…”
“Nobody needs this.”
“You don’t have enough skill to do this”

If you have similar thoughts, stop. Stop thinking in this way immediately. I had these thoughts too often but I knew what I had said at the beginning. I started this project because I wanted to train something. But I had another reason to continue working. I wanted to finish it because I had too many incomplete side projects on my mind or my hard drive. Truly, I was motivated by the answer for my first question (answer?) and willingness to complete this project. Because I did it for myself.

Many states = many problems with presentation

Designing a button which shows all its states properly is hard. We can’t freely design physical button’s state. In HTML, we have a few states to visualize (normal, hover, active, focus, visited). Each of them should look different to recognize which state is currently in use.

Focus state with normal button.

I tried to change hue, colors, and shadows to make state transitions similar to that physical buttons have. I am satisfied with the final effect but I try to find another, maybe better settings.

Hover state with filled button.

A Million colors, none of them was suitable

Colors are meaningful. It’s impossible to change the meaning of some colors e.g. red or green. We know that green color used to mean that something good happens. Appropriately, the red color usually means that something danger happened or will happen. The appearance of buttons is determined by colors and style. For me, it was the worst thing to do in this project, because I am a programmer, not a designer.

I said colors are meaningful. What happens if some (or all) colors disappear? It is an inconvenient situation for people with color vision deficiency (CVD). So, if a color doesn’t work as we expect, what else can take responsibility for giving a meaning?

Buttons with accessibility mode (with color blindness emulation).

The answer is a pattern. If we add a specific pattern to buttons with the same meaning, they still meaningful even without colors. Seems good, but how design these patterns? To be honest, I found some examples of CSS patterns and I have applied it to buttons. I haven’t done any research but in my opinion, it will be helpful.

Toolset changes dynamically

I started the project with simple Gulpfile and a few files of HTML and SCSS. Later I added nunjuck templating system but I quickly replaced them by AngularJS. It’s perfectly OK when you add or change tools to ease your work. Building the whole toolbox at the beginning doesn’t make much sense because sooner or later changes will need.

The problems arise during work. Get the right tool to solve each of them. Don’t use a hammer to everything.

I had too little time than I thought before

I had a plan to spend minimum two hours per day to push the project forward.

Not a chance.

In my case of course. I have a regular job, a private life, other hobbies which need my attention and time. Sometimes I have been done completely different things to prevent boredom. I had to change my approach to the project and I tried to do small steps every day. I defined my tasks to keep it as small as possible, so I was able to do each of them at once.

Conclusions

I know that the project which I have done is easy, small and “to-do in a few hours”. Although it wasn’t a technical challenge for me, I learned much non-technical stuff. I’m proud of myself that I did it. So, maybe you have any abandoned side project on your computer. Maybe is it a good time to refreshing it and finally finish it?