Category: www

World Wide Web

Categories
Software www

Keeping Mastodon storage in check

Mastodon storage usage on Cloudflare R2
R2 storage after aggressively clearing

For my Mastodon instance, I use Cloudflare R2; mainly for two reasons:

  • Storage was growing quickly (~80Gb during its peak); I am hosting my instance on a RPi4 (w/ 8Gb RAM) and the SSD was filling up rapidly,
  • I wanted something speedy to serve (big and cacheable) content (i.e. a CDN).

While I didn’t care much about storage any more, I still wanted to make sure it was kept in check, also for two reasons:

  • Mastodon downloads a copy of all content it says on the Fediverse, and keeps it until purged. So every instance has all the content from other instances. This could theoretically lead to you hosting illegal content and getting in trouble for it,
  • Cloudflare used to be my employer, and I have free access to R2. However, there’s always a risk they’ll disable my employee benefits one day and get me to pay for my used storage.

I run my Mastodon in a Docker instance, so your commands may vary (basically tootctl X Y is what matters). I run most of these commands once a week using systemd (except the media remover, that runs every day).

This will clear:

  • accounts (you never interacted with)
  • header files (big picture every account can upload)
  • profile pictures
  • link preview cards
  • orphaned media (uploaded media but not posted)
  • media (from other accounts)
  • statuses (from other accounts)
  • and as a bonus include updating Elasticsearch indices (which sound run every once in a while to optimise search)
/usr/bin/docker compose -f /srv/mastodon/docker-compose.yml run --rm shell tootctl accounts prune

/usr/bin/docker compose -f /srv/mastodon/docker-compose.yml run --rm shell tootctl media remove --remove-headers --days 15

/usr/bin/docker compose -f /srv/mastodon/docker-compose.yml run --rm shell tootctl media remove--prune-profiles --days 30

/usr/bin/docker compose -f /srv/mastodon/docker-compose.yml run --rm shell tootctl preview_cards remove --days 15

/usr/bin/docker compose -f /srv/mastodon/docker-compose.yml run --rm shell tootctl media remove-orphans

/usr/bin/docker compose -f /srv/mastodon/docker-compose.yml run --rm shell tootctl media remove --days 30

/usr/bin/docker compose -f /srv/mastodon/docker-compose.yml run --rm shell tootctl statuses remove --days 30

/usr/bin/docker compose -f /srv/mastodon/docker-compose.yml run --rm shell tootctl search deploy

Note that you should play with the --days X to find something that works for you (i.e.: you can scroll back in the history and still see posts/media, but not overload your storage).

I’ve included all the systemd files that’s needed here. Again, will only work in a Docker environment using the same paths as me.

The systemd files will need to be activated using something similar to this (but again, don’t blindly run these commands as it’ll likely not work):

cp *.service *.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now *.timer
systemctl list-timers | grep masto

Oh, and this is not specific to R2. This works even when storing everything locally.

I’ve written before on how to use Cloudflare CDN to protect/speed up your instance.

Categories
Software www

Kagi

It’s now been roughly 6 months I’ve switched from DuckDuckGo to Kagi. I started paying right away after roughly the one month trial.

I had not been very happy with DDG; mostly the results were very low quality (having to switch back to Google one too many times; results often lagged behind by months of publishing), many search bugs, and there’s been that thing with Bing advertising, and generally not seeing many improvements over the ~2 years I tried to use it full-time.

Kagi, on the other hand, has his search engine worked out. I know they also pull data from Bing, but also various other sources (apparently that includes Google) and their own scraper (Teclis, TinyGem), etc.

You can compare the Google and Kagi results. While not entirely the same, I don’t think Google‘s are better than Kagi‘s.

So search is really good and I very very rarely ever need to head back to Google. Only for very obscure errors with very few results, and Google almost never has a better answer.

It also adds some cool other features, like rewriting URLs.

reddit.com URLs getting rewritten to reddit.superuser.one.

For example, I run my own Libreddit, so when search results include reddit, I can now redirect to my own Reddit instance (sorry, you won’t have access).

My current rewrite rules. Rewrite AMP and reddit.

There’s a bunch of other cool stuff, like Bangs (short codes that would redirect directly to a site, for example w for wiki — so w flightradar would show this result instantly; and there’s more like hackernews, reddit, google, etc).

You can also lower priority of certain sites. Seeing too much Tiktok or Pinterest? Want to boost Hackernew results? All possible!

To make it work on iOS, you need to install a browser extension, but that works relatively well (it rewrites your search from Google to Kagi).

I was a bit apprehensive as it’s a very small team, and they are also trying to reinvent the browser space with Orion, which is a cut-throat tough market to get in. Two massive enterprises that would need a lot of funding and a lot of dev time.

Orion is based on Safari, and far from good enough to replace Firefox, but I’m actually mildly enthusiastic about this project as well.

The company is very transparent and there’s clear progress. If you want to get involved, they also have a Discord and a feedback/bug forum.

They don’t log your search queries, but show your total searches and what it costs them to handle those searches (server (GCP), API costs, etc).

I’m considering a family plan and will force Shan to switch once released.

So, all things considered, quite excited about this. Looking forward to see if they can revolutionise search. And I truly hope they make it.

The only thing I don’t like is that their logo looks too much like Google’s… 🙈

Categories
Software www

Mastodon server: R2

This is a very short post because to be honest, I didn’t figure much out myself.

My uploads/static files are now saved in R2 under its own URL (part of my enterprise zone) so that my normal caching rules and other settings are applied.

Add these to your application.env file:

3_ENABLED = "true"
S3_BUCKET = "<bucket name>"
S3_ENDPOINT = "https://<some-id>.r2.cloudflarestorage.com"
S3_ALIAS_HOST = "<connected domain>" 
S3_PERMISSION = "private"
AWS_ACCESS_KEY_ID = "<access_key>"
AWS_SECRET_ACCESS_KEY = "<secret_access_key>"

The token/API key is a bit hard to find, but it’s on the top right.

Then (re)deploy your site. 

I did set up a new server (my RPi4 started to struggle, and I guess if I'm half serious about Mastodon, I shouldn't host it at home), so I started afresh... But there's a way to migrate existing data to R2 as well, following this guide.
Categories
Errors Software www

Using Mastodon with Cloudflare

If you’re using Mastodon with Cloudflare CDN/protection and minify turned on, you’ll notice the site may look broken (after a few visits, when hitting Cloudflare cache).

Yeah, that’s not how it’s supposed to look.

And you’ll notice errors in the webdev tools similar to Failed to find a valid digest in the 'integrity' attribute, with computed SHA-256 integrity:

Failed to find a valid digest in the 'integrity' attribute for resource 'https://mastodon.yeri.be/packs/js/common-997d98113e1e433a9a9f.js' with computed SHA-256 integrity 'YgEhHmwjKL88zKfUOMt/qRulYurIuHzhn4SZC9QQ5Mg='. The resource has been blocked.
@yeri:1 Failed to find a valid digest in the 'integrity' attribute for resource 'https://mastodon.yeri.be/packs/js/locale_en-f70344940a5a8f625e92.chunk.js' with computed SHA-256 integrity '1VgpQjY/9w/fgRLw1QH2pfzqr36p3hINvg9ahpBiI2U='. The resource has been blocked.
@yeri:1 Failed to find a valid digest in the 'integrity' attribute for resource 'https://mastodon.yeri.be/packs/js/public-a52a3460655116c9cf18.chunk.js' with computed SHA-256 integrity 'onh6vHxzykkVgJkiww+OCPk0tKC48KMUD9GVJ8/LKJQ='. The resource has been blocked.

Basically, the sha256 hash doesn’t match the js or css static files.

This happens because Cloudflare minifies those files and thus the hash has been changed.

To get it to work correctly, you’ll need to create a Page Rule via Rules > Page Rules > Create Page Rule with the following info:

The page rule created; in this screenshot, the rule is still turned off.
  • URL: YourMastodonURL.com/packs/*
  • Settings: Auto Minify: off (do not select anything)
  • Rocket Loader: slider off
Details on the page rule. Save and deploy.

Don’t forget to purge your cache via the dashboard (for the Mastodon domain) via Caching > Custom Purge > Hostname > YourMastodonURL.com.

Categories
Apple Hardware Linux Misc Networking Software Windows www

Remote desktop and Wake-on-LAN

woman in gray cardigan holding black ipad outdoors
Not Shan. Also Shan rarely smiles when looking at work on her iPad... ;) Photo by Buro Millennial on Pexels.com

Shan uses her iPad a lot, but a lot of the more serious (interior design) work needs to happen on AutoCAD or Photoshop. That is just not going to work on an iPad.

When we’re travelling (read: holiday) she’s carrying an old Lenovo ThinkPad 13 (great device!) just “in case” she needs to open AutoCAD and edit something minor or read the drawings/dimensions. But honestly, most of the time that device is turned off and dead weight.

But all the above is just an excuse to “I was bored, and I wanted to test something”: can I use an old Raspberry Pi (zero W) to remotely wake her Intel NUC, and then use Tailscale to use RD on her iPad? Well, yes I can.

I completed this using:

  • Tailscale to remote desktop from anywhere to home
  • Cloudflare Tunnels, Access and DNS to have a web interface to wake the desktop
  • A Linux device that’s always on and in the same LAN, and that’ll run a PHP script.

Prep work: enable WOL

First off: enable Wake-on-LAN (WOL) in the BIOS and in your Windows settings. This article explains it for Intel NUCs, but would be similar enough for most devices. The Device Manager pane looked different on our i5 NUC, but was close enough.

On Mac, you just need to enable it in the Energy preference pane, for Linux I have no clue. 🤷‍♂️

Second step: have a working Raspberry Pi (or any Linux device) in the same LAN. This device needs to be turned on 24/7, so use something that uses very little power. 

I do have a more powerful RPi4 I wish I could've reused (running Docker and some other "serious" stuff; however it's currently in a different VLAN, and it's quite crucial the Linux device is in the same LAN as the device(s) you want to wake up), so I went with an old Raspberry Pi Zero W that was collecting dust (it used to run pwnagotchi).

On the Linux device, install etherwake. The command to run is quite simply etherwake aa:bb:cc:11:22:33 (= the ethernet MAC address of your device).

If this doesn’t wake your desktop, something is wrong and there’s no point continuing. Go and troubleshoot.

Install Tailscale and RD

On the (Windows) desktop and your iPad, install Tailscale. Login, and make sure it works by pinging from one to the other.

Then set up Remote Desktop on both (Windows, iPad). You should test and make sure you can properly connect using the LAN IP address and then the Tailscale IP address.

Fun fact: I create a DNS record for all my devices using Cloudflare DNS with the syntax of device-name.ts.yeri.be, so I don't need to ever remember IPs, and can easily ssh or ping devices without having to look up IPs.
Cloudflare DNS screenshot, filtered on “ts”. I add both A and AAAA records (not shown in the screenshot) 
Fun fact side track: I actually have a dynamic script that runs (on Linux) and creates hostname.ts.yeri.be for the Tailscale IP, hostname.wg.yeri.be based on the Wireguard IP, hostname.lan.yeri.be based on the LAN IP. This dyndns script runs every so often and updates IPs if needed. All this is running using Cloudflare DNS and their API. Super convenient.

Nginx, php and etherwake

I'm a 80s kid, so I'll use dirty PHP to run this script. I'm sure I'll go straight to hell for this, but yolo.

Install nginx and PHP (no need for MySQL and other stuff).

etherwake requires root to run (because it needs root access to create a weird magic ethernet packet). Create a file in /etc/sudoers.d/etherwake and add this line:

www-data ALL=(ALL) NOPASSWD: /usr/sbin/etherwake

This will allow www-data (nginx/php) to run /usr/sbin/etherwake using sudo, without password.

In /var/www/html/ create an index.php file with:

<html>
<head><title>Wake on Lan</title></head>
<body>
<p>Wake up <a href="mycooldesktop.php">My Cool Desktop</a>
</body>
</html>

And a mycooldesktop.php (or whatever) file with:

<?php
$output = shell_exec('sudo /usr/sbin/etherwake aa:bb:cc:11:22:33 2>&1');
echo "<pre>$output</pre>";
?>
<p><a href="..">Back</a></p>
It ain't pretty -- but it gets the job done.

And be sure to edit the MAC address to match your desktop’s ethernet MAC address.

Cloudflare tunnels

Install Cloudflare Tunnel (via Zero Trust dashboard).

When creating a new tunnel, the dashboard will give you all needed commands to install the tunnel on your RPi — but be sure to select the right OS/architecture (arm64? arm? armhf?).

Easy.

And then point the tunnel to http://localhost. No need to mess around with SSL certs.

Lastly, set up Cloudflare Access (via the same Zero Trust dashboard): create a new application, and make sure only approved users can sign in (i.e.: using a pin code emailed to only approved emails).

I only allow access to my own emails, and my family, by using groups.

You can try it yourself via wol.superuser.one. You won’t get in. 🙂

Optionally, but recommended: lock down Nginx to only allow connections from localhost (127.0.0.0/8 and ::1) if using Cloudflare Tunnels, or Cloudflare IPs if using port forwarding with Cloudflare Access in front.
Love my design skills… But it gets the job done: click that link, and it’ll execute etherwake.

To recap

  • We used Tailscale to create a VPN network between the desktop and the iPad. The big benefit is that Tailscale works effortlessly across NAT networks without having to open ports,
  • We used Cloudflare DNS so we don’t need to remember hostnames :),
  • We used Cloudflare Tunnels to make sure the RPi web interface is accessible across NAT (without port forwarding) and from anywhere,
  • We used Cloudflare Acces and locked down access to the right people using ACLs,
  • We used etherwake running on a RPi to wake up devices that are hibernating or turned off.

And that’s it really.

PS: technically WoL works with WiFi, but when I enabled WoL on the WiFi adapter, the NUC refused to hibernate/sleep for more than a minute, and kept waking itself up. So, there seems to be some kind of trigger in my network that keeps waking it up. Also, not sure if WoL via WiFi would work if the device is turned off (as opposed to sleep or hibernate). I just ended up using ethernet. 

PPS: both Cloudflare Tunnels and Tailscale use Wireguard tech in the background, so that's really cool.