Keep running and count the number of files in a folder using inotifywait
Over the last few weeks, I've been working on a Python script that generates PDFs. My script had to generate 70,000 of them and that obviously takes a while.
My idea was to have my script run in a Linux console and, in a second console, with a counter that increases with the number of files that have been created on the hard disk.
The first script will be in Python and I wanted something ultra-simple and using a simple Linux command.
The command ls folder_name | wc -l
works but didn't stay running. Let's see how we can do better with inotifywait.
Let's create our Python environmentโ
You can skip this part if you already have Python installed on your computer.
I need a Python environment so let's quickly create it thanks to Docker.
Please create a file called Dockerfile
with this content:
Dockerfile
# syntax=docker/dockerfile:1
FROM python:3.10-slim AS base
# The /app/out folder where we'll create files
RUN mkdir -p /app/out
# The /app/src where we'll put our Python script
WORKDIR "/app/src"
# Keep the container running
ENTRYPOINT ["tail", "-f", "/dev/null"]
Create the image by running docker build --tag inotify .
.
We can now create our Docker container: docker run --detach --name demo -v ./src:/app/src -v ./out:/app/out inotify
.
This will create a Docker container that will remain running. We'll share our script src/script.py
with the container and, too, the out/
folder on our host as the /app/out
folder of the container.
Create a sample Python script:โ
We need a very small Python script to generate our files:
src/script.py
import os
import signal
import tempfile
import time
from pathlib import Path
def signal_handler(sig, frame):
print('Stop')
exit(0)
signal.signal(signal.SIGINT, signal_handler)
# We'll put files in our application out folder
folder : Path = Path("/app/out/")
folder.mkdir(parents=True, exist_ok=True)
counter : int = 0
while True:
file : Path = folder.joinpath(os.path.basename(tempfile.mktemp()))
print(file.absolute())
with open(file, 'w') as f:
pass
counter += 1
print(f"{counter} files have been created.")
time.sleep(1)
Creating the monitory.sh scriptโ
Please create a script called monitor.sh
with this content:
monitory.sh
#!/bin/bash
# cspell:ignore inotifywait,inotify
count=0
# Folder to monitor
folder="$1"
# Monitor the directory for changes
# -r - Recursively monitors subdirectories
# -m - Monitor file modifications
# shellcheck disable=SC2034
inotifywait -m ${folder} | while read -r line; do
#shellcheck disable=SC2012
if (( count != $(ls "${folder}" | wc -l) )); then
count=$(ls "${folder}" | wc -l)
# Clear the terminal screen
clear
# Count the files and display the updated count
echo "Number of files in ${folder}: ${count}"
fi
done
Make the script executable: chmod +x ./monitor.sh
and make sure to install inotify by running sudo apt-get update && sudo apt-get install -y --no-install-recommends inotify-tools
.
You can quickly check if inotifywait
is already installed by running which inotifywait
. If you don't get an answer (empty response); then it's not yet there.
We're finally ready.
Running the scriptsโ
First, in a separate console, we'll start our monitoring script: ./monitor.sh out
.
In a second window, start the Python script: docker exec -it demo python script.py
.
Conclusionโ
Now, I can minimized the main screen and just keep the counter displayed.
I'm also sure that the script is well creating file since I've used two different technologies; Python and Bash.