Automatic file versioning after change using fswatch and git
I create a lot of notes. Seriously, I write tons of notes. Inspired by Getting Things Done method, I treat my mind as a thoughts generator rather than the storage. I capture thoughts, ideas, inspiring quotes, links, and pictures. Mostly using files.
Although I’m on the early stage of creating my custom note-taking solution, I’ve done some work to synchronize notes between devices and versioning them. In this article, I’d like to focus on the latter and I’m going to show you how to set up automatic files versioning.
I’m a software developer so, unsurprisingly, I use git to versioning my notes as well. Instead of manually committing changes, I take advantage of tools like fswatch and launchd to automate this process.
The problem
My notebook (if it’s a proper name for a directory with files) consists of simple text and Markdown files as well as attachments to them like pictures, archives, records, etc. The internal structure doesn’t matter.
Keeping notes as regular files instead of dedicated Notebooks (like Evernote or OneNote) let me use any tool I need at the moment or I really like. It has also some drawbacks – most of the features provided by dedicated software are missing, including the synchronization and versioning of my notes.
I decide to use git to keep track of changes in files. Moreover, I want to automate this process. I don’t want to remember to commit each change in the note, so I delegate this operation to my OS.
Initial steps
Prepare repository
Make sure that you have git installed on your OS. I use macOS, the fastest way to install any software is using homebrew.
brew install git
Then, create a dedicated directory for your notes/files and put them here. I store my versioned notebook under the path /Users/szymon/Documents/Notes.
It’s time to initialize a new repository. Using terminal, go to the newly created directory and run git init.
cd /Users/szymon/Documents/Notes
git init
The repository is ready. It’s time to configure watcher to receive information about changes in files.
Install fswatch
Actually, we don’t need to know what exactly has changed (it will be covered by git). We need to know only that change occurred in our repository.
I choose fswatch as a tool to observe changes in the directory. It’s cross-platform, works well on macOS and is able to check directories recursively. Launchd, however, is also able to watch for changes in the directory but it doesn’t track nested directories.
You can install fswatch using brew.
brew install fswatch
If it’s done, we can go to connecting these tools.
Prepare scripts to operate
One of the UNIX philosophy sounds “Make each program do one thing well”. Actually, UNIX-like systems are full of programs which do one thing: cat, grep, tail, head, xargs. The true power lies in the possibilities of connecting all these little programs together.
Let me describe what we need to achieve. For each change in the directory we need to add changed files to stage and commit them. So, we have basically 2 different operations which are connected:
- (1) Watch the directory and run command for each change
- (2) Stage and commit all files in the repository with message
Creating these 2 scripts separately gives us more possibilities. We can test them easily, we can replace one of the scripts with another one, we can reuse them in a completely different scenario, etc.
Add all to the stage and commit
First, let’s create the script to stage and commit all changes in the repository (2). I usually put scripts like this in ~/.scripts directory. You can create this file using the command touch ~/.scripts/commit-all.sh.
#!/bin/bash
git add .
git commit -m "Changed: $1"
You can also add git push command to push changes to the remote repository.
If you’re more advanced git user, you probably know, that git allows to pass option -a to commit command to stage modified files. Unfortunately, it ignores new files which are not tracked yet.
-a, -all Tell the command to automatically stage files that have been modified and deleted, but new files you have not told Git about are not affected.
The $1 is an argument to the script and it’s used in the commit message. Make this script executable by command chmod +x ~/.scripts/commit-all.sh.
Run command to observe changes
The second script runs fswatch with a set of options and passes the name of the changed file to the further execution. Start with creating script touch ~/.scripts/run-on-change.sh.
#!/bin/bash
/usr/local/bin/fswatch -e ".git" -e ".*\.swp" -e ".*~" -0 . | xargs -0 -n 1 -I {} basename {}
Similarly, we need to make this script executable by command chmod +x ~/.scripts/run-on-change.sh.
To prevent watching specific pattern I exclude them using the -e option. It’s important to exclude the .git directory – out intention is to automatically commit all detected changes. We don’t want to commit anything after changing something in the repository.
Combine scripts
We have a simple script which detects changes in the current working directory and another one which adds and commits all files to the repository. Let’s combine them together.
In your repository, run following command
~/.scripts/watch-and-commit.sh | xargs -I{} ~/.scripts/commit-all.sh {}
Now, if we change or add a file to the directory, it would be immediately committed. Moreover, the commit message would contain the name of the changed file.
Schedule execution using launchd
It’s not the first time when I use launchd as a job scheduler instead of a cron service. I briefly wrote about it in my article about synchronizing local files with encrypted vault. Basically, launchd is a recommended solution for macOS for job scheduling.
Because I use macOS on a daily basis, I focus mostly on this platform, however, we can reach the same goal using a tool called supervisord. It’s my first choice when I will configure this flow server-side or on Linux-powered machines.
I created a pl.skrajewski.notes.autocommit.plist file in directory ~/Library/LaunchAgents/ which is one of the locations for agents which work on behalf of the user.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN http://www.apple.com/DTDs/PropertyList-1.0.dtd>
<plist version="1.0">
<dict>
<key>Label</key>
<string>pl.skrajewski.notes.autocommit</string>
<key>ProgramArguments</key>
<array>
<string>bash</string>
<string>-c</string>
<string>~/.scripts/watch-and-commit.sh | xargs -I{} ~/.scripts/commit-all.sh {}</string>
</array>
<key>KeepAlive</key>
<true/>
<key>WorkingDirectory</key>
<string>/Users/szymon/Documents/Notes</string>
<key>StandardOutPath</key>
<string>/tmp/pl.skrajewski.notes.autocommit.out.log</string>
<key>StandardErrorPath</key>
<string>/tmp/pl.skrajewski.notes.autocommit.error.log</string>
</dict>
</plist>
In short: Label is an identifier of the agent. System runs the command specified in ProgramArguments in WorkingDirectory and redirects stdout and stderr to files defined as StandardOutPath and StandardErrorPath. The option KeepAlive means, that launchd will restart the job each time it goes down.
The job will start once is loaded. You can do this using launchctl.
launchctl load ~/Library/LaunchAgents/pl.skrajewski.notes.autocommit.plist
If everything is set up properly, each change in directory produces new commit. If not, we can use log and error file to investigate what goes wrong.
Weak points of this solution
This solution is far from ideal and has some weak points.
A proposed watcher, fswatch doesn’t observe changes in .git directory due to exclusion policy. However, if we checkout past commit, it will be treated as a change, so the script will be trying to commit changes. If you want to restore the previous version of files, make sure that the agent is not running. You can unload it using the following command.
launchctl unload ~/Library/LaunchAgents/pl.skrajewski.notes.autocommit.plist
Nevertheless, it’s better to change the script a bit to avoid committing if e.g. HEAD is from master branch. It’s a field to improve.
The problems may come also from the editor you use. Some of the editors store temporary files next to original files and they may trigger auto-commit each time when the content of file change. You’ll have to exclude these files in the fswatch invocation.
Additionally, a lot of editors have auto-save functionality, creating in outcome lots of commits with relatively small differences. Instead of committing changes immediately, we can delay this operation a bit. If something change before the commit, the script will cancel the previous invocation and it’ll schedule the commit with new changes.
Summary
Although the whole process sounds like reinventing the wheel, I hope that you found out something interesting for you. You can treat this article like a proof-of-concept or a presentation of the idea rather than the complete recipe.
I used to write software which does something I needed. Sometimes, I spent days or weeks before I got something useful. Recently, I’ve discovered the real power of combining small and simple tools, mostly provided by my operating system. It just works.
It doesn’t mean that writing software is bad. In my opinion, writing is the best way to learn. But the main goal of software should be to solve a problem. If the problem can be solved using tools we already have, why not use them? At least until we want to generalize the use case to be useful for others.
