Skip to main content

Exterminate them all, kill spam using GitHub Actions

ยท 8 min read
Christophe

Exterminate them all, kill spam using GitHub Actions

In previous articles, I explained how to fight against spam if you've a cpanel or PlanetHoster's N0C infrastructure.

What if we automated as many things as possible?

In the PlanetHoster's N0C infrastructure article , I showed that you need to write rules for RoundCube to identify sources of spam and remove them. These rules have to be created from the RoundCube interface and, well, ok, that's not optimized at all.

Let's look at how to do as little as possible and still be a hardcore spam fighter.

The ideaโ€‹

As already explained, RoundCube is using a sieve file to store your rules. Such file can look like this:

# rule:[Identify as spam: *.su]
if allof (header :matches "from" "*.su")
{
fileinto "spam";
}

And, therefore, the objective is simple and obvious: we're going to create a little script that only needs two things.

  1. a list of values, e.g. domain names to be blocked
  2. a sieve template

Our action planโ€‹

  1. We'll create a directory and jump in it: mkdir ~/sieve-generator && cd $_,
  2. We'll create our JSON file,
  3. We'll create our sieve template,
  4. We'll create our script, in Bash (but you can do this in any language)
  5. We'll create a repository on GitHub and push our code there
  6. And we'll finally set up GitHub Actions to automate the build and deploy process.

Createโ€‹

Please run mkdir ~/sieve-generator && cd $_ to create the sieve-generator in your home directory and jump in it.

Storing the list of values in a JSON fileโ€‹

Please create a file called patterns.json with this content:

{
"spam": [
"*.buzz",
"*.cf",
"*.cn",
"*.ga",
"*.gq",
"*.host",
"*.icu",
"*.india",
"*.info",
"*.live",
"*.ml",
"*.net",
"*.online",
"*.ru",
"*.su",
"*.tk",
"*.top",
"*.ua",
"*.us",
"*.wang",
"*.xyz"
]
}

Create our templateโ€‹

Please create a file called spam.template with this content:

# rule:[Identify as spam: {{ pattern }}]
if allof (header :matches "from" "{{ pattern }}")
{
fileinto "spam";
}

Please keep the last empty line

The last line of the spam.template file has to be an empty line so the file should have 6 lines, not 5.

Create the generation scriptโ€‹

Please create a file called generate.sh with this content:

#!/usr/bin/env bash

clear

if [ ! -f patterns.json ]; then
echo "Please create a patterns.json file."
echo "Run 'cp patterns.json.dist patterns.js' to use an example."
exit 1
fi

if [ ! -f spam.template ]; then
echo "Please create a spam.template file. Below an example of what you can put in it:"
echo ""
echo "# rule:[Identify as spam: {{ pattern }}]"
echo "if allof (header :matches \"from\" \"{{ pattern }}\")"
echo "{"
echo " fileinto \"spam\";"
echo "}"
echo ""
exit 1
fi

# File to create
outFile="${PWD}/build/roundcube.sieve"

[ ! -d "$(dirname "${outFile}")" ] && mkdir "$(dirname "${outFile}")"

rm -f "${outFile}" && touch "${outFile}"

cat patterns.json \
| jq '.spam[]' \
| sort \
| while read -r pattern; do \
# Trim quotes
pattern=$(echo "$pattern" | tr -d '"')
# Read the spam.template file, make the replace and append in file roundcube.sieve
sed "s/{{ pattern }}/${pattern}/g" spam.template >>"${outFile}"; \
done

echo "File "${outFile}" has been created."

Also run chmod +x generate.sh to make it executable.

At this stage, you've thus three files in your folder:

  • generate.sh,
  • patterns.json and
  • spam.template

In your console, run ./generate.sh and if everything is running fine, you'll get a message like File /home/xxx/sieve-generator/build/roundcube.sieve has been created..

If you get an error about jq please install it by running: sudo apt-get update && sudo apt-get install jq then start ./generate.sh again.

So now, you should have a new folder called build where the file roundcube.sieve has been created. Open the file and check his content. It'll start with:

# rule:[Identify as spam: *.buzz]
if allof (header :matches "from" "*.buzz")
{
fileinto "spam";
}
# rule:[Identify as spam: *.cf]
if allof (header :matches "from" "*.cf")
{
fileinto "spam";
}
# rule:[Identify as spam: *.cn]
if allof (header :matches "from" "*.cn")
{
fileinto "spam";
}

Congratulations: you've successfully created your anti-spam generator.

At this stage, you can deploy the file to your FTPโ€‹

If you wish to validate the roundcube.save file, you can start your FTP client, go to your N0C server (hosted by PlanetHoster), go to the folder /mail/DOMAIN.TLD/ACCOUNT/sieve (refers to the article PlanetHoster's N0C infrastructure if needed).

Push the file there, on your host.

Now, go to your web email account (direct link: https://mg.n0c.com/email/accounts) and start RoundCube. Open your account, click on Settings then Filters and tadaaa...

RoundCube filters

I've already added more spams domain for my own account

If you don't have the same list of filters; it's absolutely normal.

Create your own Github repositoryโ€‹

Skip this chapter if you don't want full automation

In order to be able to create automation using Github Actions, first, we need to create a Github repository.

I'll suppose you already have a Github account and you know how to use Github.

Go to https://github.com/new and create a new private repository (it's free).

Then just copy/paste the example given by Github; something like:

echo "# Sieve generator" >> README.md
git init
git add README.md
git commit -m "first commit"
git branch -M main
git remote add origin XXXXXXXXXXXX '<-- put here your own path
git push -u origin main

Now push the other files:

git add .
git commit -m "Initial upload"
git push

Back to github, you'll obtain a repo like this one:

Github repo

So, now, each time you'll update the list of domains (in the patterns.json file), our objective is to ask Github to run itself the ./generate.sh script to generate the build/roundcube.sieve file.

And, too, to publish the new file to your FTP account so, to put it simply: you modify the JSON file on your computer and RoundCube receives the new rules automatically, just a few seconds later.

Adding Github Actionsโ€‹

Skip this chapter if you don't want full automation

Go back to your computer, jump back in the ~/sieve-generator folder and create a new file called .github/workflows/build_and_deploy.yaml.

Here is the content for that new file:

name: Create roundcube.sieve and deploy

on:
push:
branches:
- main

workflow_dispatch:
inputs:
version:
type: string
required: false
description: 'Version of jq to install'
default: '1.7'
force:
type: boolean
required: false
description: 'Do not check for existing jq installation before continuing.'
default: false

permissions:
contents: write

jobs:
deploy:
name: Create roundcube.sieve and deploy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: 'Setup jq'
uses: dcarbone/install-jq-action@v2
with:
version: '${{ inputs.version }}'
force: '${{ inputs.force }}'

- name: 'Build'
run: |
./generate.sh

- name: Push files
uses: SamKirkland/FTP-Deploy-Action@v4.3.4
with:
server: ${{ secrets.ftp_server }}
username: ${{ secrets.ftp_login }}
password: ${{ secrets.ftp_password }}
local-dir: ./build/

I won't go into detail about this file, but this ask Github to:

  1. Install JQ on his server,
  2. Run the generate.sh script and thus build the build/roundcube.sieve file
  3. Use FTP-Deploy-Action to push the file on your server.

As you can see, there are three secrets.

Back to your browser, go to your Github repository you've created in the previous chapter.

Click on Settings then Secrets and variables and finally, click on Actions

Github secrets

Click on the green button New repository secret and create the first one called FTP_SERVER (seems to be in uppercase). Put there the name of your FTP server like node99-xx.N0c.com (get the name from your My planethoster page).

You'll need to add two new secrets FTP_LOGIN and FTP_PASSWORD but, before, create a new FTP user.

Go to https://mg.n0c.com/files/ftp-accounts and create a new user. Use a very explicit name like deploy_spam_roundcube_xxx and define a very strong password (30 chars or more).

Restrict the path for that user to, exactly, where the sieve file is located on your FTP. As said in PlanetHoster's N0C infrastructure, for me, it's /mail/avonture.be/christophe/mail/avonture.be/christophe/sieve. Make sure to find the path for you.

note

In case the sieve folder didn't exists yet, the best way to create it, I think, is to manually create a filter in Roundcube. Here is the official PlanetHoster documentation.

Ok, so, now, you've a new FTP login, his very secure password and being able to, only, connect to your sieve folder.

Copy the login and the password in the Github secrets page.

You'll then have this:

Secrets

Everything is now in place for automationโ€‹

If you've followed the Github chapters, now, everything is set up.

Back to your computer and edit the patterns.json file and add a new pattern, f.i. newsletters@*.

Push the change to Github (git add . ; git commit -m "chore: new pattern" ; git push).

Just for the curiosity, go to Github - Actions tabs:

Build and deploy

If you've configured everything correctly, you should get this:

FTP Deploy Action

Conclusionโ€‹

From now on, all you have to do is copy/paste the email addresses you receive spam from, see if you can standardise them as much as possible (by using the asterisk like in newsletter@*) and then copy/paste them into the JSON file that you just have to upload to Github.

Et voilร !