Odi's astoundingly incomplete notes
New Entries
Restic is my favorite backup tool
Restic deserves a shoutout. Only recently I have discovered this outstanding backup solution. It's simplicity is ingenious. It allowed my to replace my 10 year old Bacula solution in just 15 minutes! Now I have a reliable automate backup with no clicky UI, no central database that uses minimal resources.
The coolest feature is its git-like storage. Files are only stored once. There is no need to distinguish between full, incremental or differential backups. Every backup is a full backup, but stores files efficiently only once. That makes backup management a dream.
Restic's key features that make it so cool:
The coolest feature is its git-like storage. Files are only stored once. There is no need to distinguish between full, incremental or differential backups. Every backup is a full backup, but stores files efficiently only once. That makes backup management a dream.
Restic's key features that make it so cool:
- No database, no central infrastructure: the backup is the database.
- No configuration necessary: just decide where to put the backup and choose a passphrase. Everything else is optional and honestly not really necessary.
- High throughput by using parallel compression/encryption and no database limits what you can do: I was genuinely blown away by the speed.
- Deduplication and efficient storage: git-like storage, no differential backups necessary
- No connectivtiy woes: since there is no central component
- Works on local disk as well as almost any cloud storage protocol: unlike Bacula which is tape library centric and drive backups are a side feature
- Small single binary (go)
- Highly scriptable: the binary has a couple of subcommands to manage everything. Takes arguments or env vars. Provies exit codes.
- No UI: the single binary is the cli and the backup tool itself
- Encryption by default: you can't disable it, but if you don't like it simply select a trival password that you put next to your backup.
Add comment
Creating a kernel source diff
When you roll your own kernel and frequently update it to the current stable version from
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git, then this script may be helpful. It creates the diff from the currently running kernel version to the latest stable one. You can easily apply it to your build source tree with: patch -p1 <~/patch.diff
#!/bin/bash
REPO=~/code/linux-stable
OUT=~/patch.diff
cd "${REPO}" || exit 1
[ -d .git ] || {
echo "${REPO} is not a git repo" >&2
exit 1
}
CUR=$(uname -r)
MAJ=$(echo "${CUR}"| sed 's/\\.[^.]*$//')
MIN=$(echo "${CUR}"| sed 's/[0-9]\\+\\.[0-9]\\+\\.//')
if [ "${MIN}" = "0" ]; then
# strip .0
CUR="${MAJ}"
fi
BR=$(git branch --show-current)
echo "Using git repo: ${REPO}"
echo "Current kernel: ${CUR}"
echo "Major version: ${MAJ}"
echo "git branch: ${BR}"
if [ "${BR}" != "linux-${MAJ}.y" ]; then
echo "linux-stable on branch ${BR} which is not compatible with kernel ${CUR}" >&2
EX=$(git branch | fgrep linux-${MAJ}.y)
if [ -z "${EX}" ]; then
echo "Please do: git checkout -t origin/linux-${MAJ}.y" >&2
else
echo "Please do: git checkout linux-${MAJ}.y" >&2
fi
exit 1
fi
# starts with v
NEXT=$(git tag --sort='-v:refname' | egrep "v${MAJ}" | head -1)
if [ "${NEXT}" = "v${CUR}" ]; then
# are we uptodate at all?
echo "Pulling git"
git pull -q
NEXT=$(git tag --sort='-v:refname' | egrep "v${MAJ}" | head -1)
fi
echo "Next version: ${NEXT}"
if [ "${NEXT}" = "v${CUR}" ]; then
echo "Kernel is uptodate, nothing to do" >&2
exit 1
fi
echo "Creating diff v${CUR}..${NEXT}"
git diff "v${CUR}..${NEXT}" >"${OUT}"
echo "Done: ${OUT}"Improve font rendering in Firefox
If fonts in Firefox look uglier than in Chrome then probably LCD Filtering is not enabled in fontconfig.
In Gentoo enable LCD filtering like so:
In Gentoo enable LCD filtering like so:
eselect fontconfig enable 11-lcdfilter-default.conf eselect fontconfig listAnd make sure the other LCD filtering options are disabled. In my setup I get good fonts with:
Available fontconfig .conf files (* is enabled): [1] 10-autohint.conf * [2] 10-hinting-full.conf [3] 10-hinting-medium.conf [4] 10-hinting-none.conf [5] 10-hinting-slight.conf [6] 10-no-sub-pixel.conf [7] 10-scale-bitmap-fonts.conf * [8] 10-sub-pixel-bgr.conf [9] 10-sub-pixel-rgb.conf [10] 10-sub-pixel-vbgr.conf [11] 10-sub-pixel-vrgb.conf [12] 10-unhinted.conf [13] 11-lcdfilter-default.conf [14] 11-lcdfilter-legacy.conf [15] 11-lcdfilter-light.conf [16] 20-unhint-small-dejavu-sans.conf [17] 20-unhint-small-dejavu-sans-mono.conf [18] 20-unhint-small-dejavu-serif.conf [19] 20-unhint-small-vera.conf * [20] 25-unhint-nonlatin.conf [21] 30-metric-aliases.conf * [22] 40-nonlatin.conf * [23] 45-generic.conf * [24] 45-latin.conf * [25] 49-sansserif.conf * [26] 50-user.conf * [27] 51-local.conf * [28] 57-dejavu-sans.conf [29] 57-dejavu-sans-mono.conf [30] 57-dejavu-serif.conf [31] 60-generic.conf * [32] 60-latin.conf * [33] 60-liberation.conf [34] 63-source-pro.conf [35] 65-fonts-persian.conf * [36] 65-khmer.conf [37] 65-nonlatin.conf * [38] 66-noto-mono.conf * [39] 66-noto-sans.conf * [40] 66-noto-serif.conf * [41] 69-unifont.conf * [42] 70-no-bitmaps.conf [43] 70-yes-bitmaps.conf [44] 75-yes-terminus.conf [45] 80-delicious.conf * [46] 90-synthetic.conf *
Stackoverflow can break your locking scheme
Simple Java locking code, looking innocent, right?
Here is a little demo:
The output is something like this:
private ReentrantLock lock = new ReentrantLock();
private void doIt() {
boolean locked = lock.tryLock();
try {
...
} finally {
if (locked) lock.unlock();
}
}
Well, only until you realize that a call to unlock() causes 4 nested method calls. So unlock() requires a little bit of free stack. But what happens when stack is tight? Well... unlock() will simply fail with a StackOverflowError of course and your lock is leaked. Here is a little demo:
public class StackOfl implements Runnable {
public static void main(String[] args) {
new StackOfl().run();
}
@Override
public void run() {
try {
test();
} catch (Throwable e) {
e.printStackTrace();
}
System.out.println("still locked? "+ lock.isLocked());
System.out.println(lock.toString());
}
private void test() {
overflowStack();
}
private ReentrantLock lock = new ReentrantLock();
private void overflowStack() {
boolean locked = lock.tryLock();
try {
overflowStack(); // 1. StackOverflowError occurs here
} finally {
if (locked) lock.unlock(); // 2. StackOverflowError occurs within here
}
// avoid tail recursion optimization in compiler
System.out.println("unreachable");
}
}
This code overflows the stack on purpose within the try/finally block. Java dutifully executes the finally block on the first StackOverflowError just to run into another StackOverflowError during unlock(), which is the one that propagates up.The output is something like this:
java.lang.StackOverflowError at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:149) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261) at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457) at StackOfl.overflowStack(StackOfl.java:39) at StackOfl.overflowStack(StackOfl.java:37) at StackOfl.overflowStack(StackOfl.java:37) ... still locked? true java.util.concurrent.locks.ReentrantLock@7852e922[Locked by thread main]You can clearly see the lock is leaked.
Gentoo updates su binary
If your Gentoo spits out cron errors like this:
The problem arises since the
run-parts: /etc/cron.daily/man-db exited with return code 1after updating to
sys-apps/util-linux-2.37.3 then you should run pwck to check the consistency of /etc/passwd, /etc/shadow etc.# pwck user 'adm': directory '/var/adm' does not exist user 'lp': directory '/var/spool/lpd' does not exist user 'news': directory '/usr/lib/news' does not exist user 'uucp': directory '/var/spool/uucppublic' does not exist no matching password file entry in /etc/shadow add user 'man' in /etc/shadow? y no matching password file entry in /etc/shadow add user 'smmsp' in /etc/shadow? n no matching password file entry in /etc/shadow add user 'portage' in /etc/shadow? y no matching password file entry in /etc/passwd delete line 'games:*:9797:0:::::'? y no matching password file entry in /etc/passwd delete line 'guest:*:9797:0:::::'? y pwck: the files have been updatedOn all my machines the
man and portage users were not in shadow. And you can userdel smmsp if you are not using qmail too.The problem arises since the
su binary from sys-apps/shadow has been replaced with the util-linux version (see su USE flag).Thanks for spelling it out. Current latest stable sys-apps/man-db-2.9.4-r1 doesn't seem to be requiring new-enough sys-apps/util-linux-2.37.4, and man-db cron job starts failing every day.
Gentoo switches to libxcrypt
Update these packages before others:
emerge -v1 virtual/libcrypt sys-libs/libxcryptOtherwise you can run into dependency issues if there are a lot of other package updates. Also this gets you into a stable state again (as it causes a glibc rebuild) more quickly when you update a system that can't afford longer downtimes.
Cast CLOB to BLOB in Oracle
select TO_BLOB(UTL_RAW.CAST_TO_RAW(myclob)) from mytableThis will provide a CLOB column as a BLOB. The character set / encoding that is used here is to be determined. Probably DB default.
Gentoo switches to Python 3.9
To find remaining packages that still have a python3_8 useflag that prevents depcleaning 3.8:
find /var/db/pkg -name USE | xargs grep python3_8Gentoo has a binary Rust package
Rust takes a lot of time and resources to compile. Very annoying.
Fortunately there is a binary package: dev-lang/rust-bin
But virtual/rust prefers the source package via
Fortunately there is a binary package: dev-lang/rust-bin
But virtual/rust prefers the source package via
RDEPEND="|| ( ~dev-lang/rust-${PV}[${MULTILIB_USEDEP}] ~dev-lang/rust-bin-${PV}[${MULTILIB_USEDEP}] )"
So any sane person should add dev-lang/rust to their /etc/portage/package.mask file, so portage picks rust-bin by default.Root can NOT access all files
If you think that root has universal privilege on the filesystem, then you may need to update your opinion.
Consider an NFS share for example that is exported from the server as:
and the permissions on that directory are:
As the man page for exports explains:
Very often, it is not desirable that the root user on a client machine is also treated as root when accessing files on the NFS server. To this end, uid 0 is normally mapped to a different id: the so-called anonymous or nobody uid. This mode of operation (called 'root squashing') is the default, and can be turned off with no_root_squash.
So root being able to access almost everything is only true for "normal" filesystems, but a network filesystem or any other form of mount (fuse!) may behave differently althogether.
Beware of that fact when doing backups. Running backup as root may not be sufficient.
Consider an NFS share for example that is exported from the server as:
/var/nfs 10.1.2.3(rw)
and the permissions on that directory are:
drwxrws---. 2 foo bar 6 Apr 15 17:33 nfsOn the client where that NFS share is mounted the directory looks innocuous (same as above). But when root tries to enter it
# cd /var/nfs bash: cd: /var/nfs: Permission deniedWhat happens here? The NFS share uses root squashing, which will map the client's root user to the local anonymous user.
As the man page for exports explains:
Very often, it is not desirable that the root user on a client machine is also treated as root when accessing files on the NFS server. To this end, uid 0 is normally mapped to a different id: the so-called anonymous or nobody uid. This mode of operation (called 'root squashing') is the default, and can be turned off with no_root_squash.
So root being able to access almost everything is only true for "normal" filesystems, but a network filesystem or any other form of mount (fuse!) may behave differently althogether.
Beware of that fact when doing backups. Running backup as root may not be sufficient.