Mutexes in Linux shells#

Linux shells do not have built-in mutexes. However, it is possible to simulate them using the following approaches:

1. Implementation With mkdir#

The mkdir command in Linux is atomic on POSIX-compliant local file systems. This means that creation of a new directory either completes successfully or fails, and there is no intermediate state.

If two or more processes attempt to create the same directory simultaneously, only one will succeed, and the rest will receive an error. This makes the mkdir command a suitable and portable way for creating locks.

Before entering the critical section, the process checks if the lock directory exists. If the directory does not exist, then it means no other process is in the critical section or accessing the protected resource.

The process can then attempt to create the lock directory. If the directory is created successfully, then the lock is obtained and the process can enter the critical section. If there is an error creating the directory, then another process must be entering or has entered the critical section. The process can then choose to wait or abort execution.

Once the process is done, it can delete the lock directory to release the lock. A trap can also be set to clean up the directory in case the script exits cleanly.

1
LOCK_DIR="/tmp/lock_file"
2

3
if ! mkdir "$LOCK_DIR" 2>/dev/null; then
4
    # resource is already locked
5
    # : exit or wait for it to be unlocked
6
    echo "Another processes has the lock."
7
    exit 1
8
fi
9

10
trap 'rmdir "$LOCK_DIR"' EXIT
11
# lock acquired on resource
12
# enter your critical section here...

2. Implementation With flock#

flock is a Linux utility used to manage advisory locks on open files from within a shell script or the commandline. The utility helps to achieve synchronized access to a file protecting its contents from accidental corruption. It relies on the underlying flock() system call, which associates the lock with the file descriptor a process holds for an open file.

The lock is an advisory lock and thus it is not enforced by the kernel. This means that a process can still have access to the file even if it is locked. All processes that access your resource must use flock and respect the lock for it to work.

It can be used to obtain shared or exclusive access to a file, wait on a lock on a file, or release a lock on a file. The default behavior of the utility is an exclusive lock. Any locks not released are automatically released when the process ends.

NOTE

The lock file can be either a separate lock file or the actual data file to be protected by the mutex.
A personal recommendation is to use the data file as the lock file unless your requirements need otherwise, such as when protecting a directory. This way, you have fewer file system footprints and also simplify your script logic.

1
# Exclusivley locks the file data.txt for writing
2
flock data.txt -c '
3
    DAT=$(cat data.txt);
4
    echo $((DAT+1)) > data.txt;
5
    cat data.txt
6
'
7

8
# Obtain a shared lock on data.txt for reading
9
flock -s data.txt -c 'cat data.txt'

1
# In this example the data file is also the lock file
2

3
# open the data file and assign it the file descriptor number 300
4
DATA_FILE='data.txt'
5
exec 300 > "$DATA_FILE"
6

7
# obtain an exclusive lock on the file for writing. We use -n to overide default blocking behaviour
8
if ! flock -n 300; then
9
    echo "The DATA FILE is already locked. Exiting."
10
    exit 1
11
fi
12

13
# The exclusive lock was obtained so we can enter critical section
14
echo "In critical section"
15

16
# ---------------------
17
# if the script terminates, the file descriptor will be closed and the lock released automatically.
18
# ---------------------
19

20
# if the script continues with further processing after the critical section, you can manually unlock the file or close the FD as follows:
21

22
### This only releases the lock on the file. The file descriptor is still open and can be used later.
23
flock -u 300
24

25
### This closes the file descriptor in which case the lock is automatically released.
26
exec 300 > &-

Example Use for Mutexes#

We’ll create a simple script to demonstrate mutexes in use in Linux.

1
/data/dir_0
2
/data/dir_1
3
/data/dir_2
4
/data/dir_3
5
/data/dir_4

1
#!/bin/bash
2

3
DATA_FILE="dirs.txt"
4
DATA_FILE_FD=200
5

6
# open data file and assign a file descriptor
7
eval "exec $DATA_FILE_FD<>\"$DATA_FILE\""
8

9
# function to attempt to the lock data file 3 times
10
function Lock_Data_File {
11
    local max_attempts=3
12
    local attempt=1
13

14
    while [ $attempt -le $max_attempts ]; do
15
        # wait for the lock to be available for 2 seconds
16
        if flock -w 2 $DATA_FILE_FD; then
17
            return 0
18
        fi
19
        ((attempt++))
20
    done
21

22
    echo "Failed to lock data file after 3 attempts..!"
23
    return 1
24
}
25

26

27
# get initial directory to process
28
if ! Lock_Data_File; then
29
    exit 1
30
fi
31

32
LINE_INDEX_TEXT=$(awk '!/.+:.+/ {printf "%d:%s\n", NR, $0; exit}' "$DATA_FILE")
33

34
while true; do
35
    # if no more data to process, exit, this will release the lock and file decriptor as well
36
    if [ -z "$LINE_INDEX_TEXT" ]; then
37
        echo "No more directories to process"
38
        exit 0
39
    fi
40

41
    LINE_INDEX=`echo "$LINE_INDEX_TEXT" | cut -d: -f1`
42
    LINE_TEXT=`echo "$LINE_INDEX_TEXT" | cut -d: -f2`
43

44
    # mark the dir as processing and unlock data file
45
    sed -i "${LINE_INDEX}s/^/P:/" "$DATA_FILE"
46

47
    flock -u $DATA_FILE_FD
48

49
    # process the dir, sleep for 10s to simulate long process
50
    echo Processing $LINE_TEXT
51
    sleep 10
52
    echo Done Processing $LINE_TEXT
53

54
    # lock the data file, mark the dir as processed and get next directory to process
55
    if ! Lock_Data_File; then
56
        exit 1
57
    fi
58

59
    sed -i "${LINE_INDEX}s/^P:/C:/" "$DATA_FILE"
60

61
    LINE_INDEX_TEXT=$(awk '!/.+:.+/ {printf "%d:%s\n", NR, $0; exit}' "$DATA_FILE")
62
done

Summary#

• Linux shells don’t provide built-in mutex implementations.
• It is possible to simulate mutex behavior using file operations, specifically the mkdir and flock commands.
• The mkdir command is limited in portability and also may not work on all file systems.
• The flock command is more robust and portable.
• The flock command and its associated flock system call use advisory locks. These are not enforced by the kernel.
• With the flock command, the worker processes must respect the advisory locks for the mutex behavior to work.

Conclusion#

Unless you have a good reason, prefer the flock command for mutex implementations in Linux shell scripts. This is more portable than using the mkdir command.