shuf on Mac: sort -R Kept Duplicates Together 2,000/2,000

In August I wrote one sentence about sort -R in a post about timeout command not found on Mac: it is not shuf, because identical lines travel together. I based that on two runs of a five-line file. That is thin evidence for something people paste into scripts, and "shuf mac" gets 83 Google autocomplete completions, including "mac shuf command not found" and "shuf install mac". So today I measured it properly on this Mac mini (macOS 26.4.1) against GNU shuf 9.12, and checked the other stock substitutes people suggest.

One line from my setup first, because it matters for testing. Homebrew coreutils went onto this machine on September 29, and since Homebrew only adds the g prefix to commands macOS already ships, shuf is installed under its plain name at /opt/homebrew/bin/shuf, next to gshuf. A script that calls shuf passes on this Mac and fails with shuf: command not found on a stock one. Every test below pins PATH to the four system directories and calls GNU shuf by absolute path.

Why shuf is missing on macOS

shuf is a GNU coreutils program, not a POSIX one, and macOS ships BSD userland. Of the 109 coreutils programs, 19 have no command of the same name on a stock Mac, and shuf is one of them (the full list is in the timeout post). The common fixes are:

• brew install coreutils, which today gives you both shuf and gshuf. Older answers say only gshuf exists. On 9.12 the formula's caveat says the prefix applies only to "commands also provided by macOS" plus dir, dircolors, and vdir, and 13 programs, shuf and timeout among them, are installed unprefixed.
• Replace it with something on the stock system. Mostly that means sort -R, an awk one-liner, jot -r, or perl. These are where the bugs are.

sort -R groups duplicate lines, every time

The Mac sort man page says it plainly: -R is "a random permutation of the inputs except that the equal keys sort together. It is implemented by hashing the input keys and sorting the hash values." Equal lines get equal hashes, so they always come out next to each other.

I shuffled a six-line file (a b b c c d) 2,000 times with each tool and counted runs where both duplicate pairs came out adjacent:

sort -R                  2000 / 2000
shuf (GNU 9.12)           261 / 2000
expected for a shuffle   ~267 / 2000   (4! x 2 x 2 / 6! = 13.3%)

The same thing happens with keys. With sort -R -k1,1, lines that share a first field stay together: in four runs over x 1, x 2, y 1, y 2, z 1, the two x lines and the two y lines were adjacent every time.

This is how it gets into real code. A merged stellar-core PR from 2021, #2866, says "shuf is not always available on Mac, but sort -R seems to be more universal." For a list of unique test names, that's fine. For anything with repeated lines, it isn't a shuffle.

sort -R | head -1 picks the wrong line

The worse case is picking one random line, which is what most "shuf on Mac" searches want. A 2022 issue on a Mac wallpaper changer, KindaOK/MacDesktopChanger#1, suggests replacing shuf with sort -R | head -n 1. Because duplicates hash together, that command picks uniformly among the distinct lines and ignores how often each one appears.

I ran it on a file with a once and b three times. A fair pick returns a a quarter of the time. Over 4,000 draws per method:

shuf -n 1 returned a 949 times. sort -R | head -1 returned it 1,941 times. If the file is a weighted list, say a playlist where a favorite appears three times, the weighting silently disappears.

The awk shuffle repeats within the same second

The other common substitute is decorate, sort, undecorate with awk:

awk 'BEGIN{srand()} {print rand() "\t" $0}' file | sort -n | cut -f2-

POSIX says srand() with no argument uses the time of day, and on the Mac's awk (version 20200816) that means whole seconds. I checked: the seed it set was the same number date +%s printed. I ran the line 300 times back to back. It took two seconds and produced exactly two distinct orders. Leave out srand() and you get the same order on every run.

That's harmless for something that runs once a day. A merged PR in nhangen/claude-ceo#2 uses this exact line to pick three daily entries, which is fine at that rate. It breaks when a loop or a test harness calls the shuffle several times per second.

zsh hands every subshell the same $RANDOM

The obvious fix is to seed awk from the shell: awk -v seed=$RANDOM 'BEGIN{srand(seed)} ...'. Under bash 3.2 that works: 300 runs gave 300 different orders. Under zsh 5.9, the default Mac shell, the same loop gave one order 300 times.

The zsh manual explains why, in man zshparam: RANDOM is "an intentionally-repeatable pseudo-random sequence; subshells that reference RANDOM will result in identical pseudo-random values unless the value of RANDOM is referenced or seeded in the parent shell" in between. Every element of a zsh pipeline except the last runs in a forked subshell, so the $RANDOM in awk -v seed=$RANDOM ... | sort is expanded in the child, and the parent's generator never moves. The plainer version shows it too:

% for i in 1 2 3 4 5; do echo -n "$(echo $RANDOM) "; done
24497 24497 24497 24497 24497

Reading $RANDOM in the parent shell first fixes it: seed=$RANDOM on its own line, then awk -v seed=$seed. That gave 298 distinct orders in 300 zsh runs. The two repeats are expected collisions, since $RANDOM only has 32,768 values. The zsh/random module that would provide SRANDOM isn't shipped. Loading it fails with /usr/lib/zsh/5.9/zsh/random.so (no such file).

What to use on a stock Mac

These are the replacements that held up in the same tests:

Job	GNU	Stock macOS	Measured
Shuffle lines	shuf file	awk '{print NR "\t" $0}' file | sort -R | cut -f2-	263/2000 adjacent (fair); 1M lines in 0.90 s vs 0.04 s for shuf
Shuffle lines, fast	shuf file	perl -MList::Util=shuffle -e 'print shuffle <>' file	0.15 s for 1M lines; merges the last line if the file has no final newline
One random line	shuf -n 1 file	awk -v n="$(jot -r 1 1 "$(awk 'END{print NR}' file)")" 'NR==n' file	991/4000 for the 1-in-4 line
Random integers	shuf -i 1-6 -n 5 -r	jot -r 5 1 6	60,000 draws, each face 9,878 to 10,135
N distinct integers	shuf -i 1-10 -n 3	jot 10 | sort -R | head -3	safe: the lines are unique

Numbering every line is what fixes sort -R: once each key is unique, nothing hashes together. The numbered version is ten times slower than shuf on a million lines and still under a second. I also sorted its output back with sort -n and compared it with the input, and it matched, so no line is dropped or duplicated. The one-line pick is written that way on purpose. My first version was sed -n "$(jot -r 1 1 $(wc -l < file))p" file, which got 998 of 4,000 right on the test file. But wc -l counts newlines, so a file whose last line has no newline reports one line short. On a b c without a final newline, wc -l said 2, and 600 draws returned a 319 times, b 281 times, and c never. Counting with awk 'END{print NR}' and printing with awk 'NR==n' gave 225/177/198 on the same file, and awk adds the missing newline to the output.

Two more porting traps. If you want a reproducible shuffle, Mac sort rejects seed files over 32 bytes (random seed is too large (64 > 32)!, exit 64), and --random-source=/dev/urandom, the usual GNU idiom, exits 64 with "random seed is a character device other than /dev/random". Both checks are in Apple's sort.c. Going the other way, sort -R | head is quiet on a Mac but prints sort: write failed: 'standard output': Broken pipe under GNU sort, as dxw/dalmatian-tools#532 found. Add pipefail and that becomes a failure, the same kind of exit-status surprise I wrote up in bash pipe exit code.

If you control the machines, installing coreutils and calling shuf is simplest. Just test the script with PATH=/usr/bin:/bin:/usr/sbin:/sbin before shipping it, because a Mac with Homebrew hides the missing command. It's the same split behind head: illegal line count and date: illegal option -- d: the Mac command exists, takes the same name, and does something different.

Update (2026-10-01): nproc is another coreutils program that Homebrew now installs unprefixed, and its absence is even quieter than shuf's: make -j$(nproc) just runs with no job limit. Measurements and a guarded replacement are in nproc on Mac.

Sources: all measurements were run on 2026-10-01 on a Mac mini (macOS 26.4.1, build 25E253) with PATH pinned to the system directories: /usr/bin/sort 2.3-Apple, awk 20200816, bash 3.2.57, zsh 5.9, perl 5.34 with List::Util 1.55, against GNU shuf 9.12 from Homebrew. Draw counts are 2,000 runs for adjacency, 4,000 for the single-line pick, 300 for the seed tests, and 60,000 for jot. The scripts are kept with my research notes. The sort behavior is quoted from the macOS man page and Apple's text_cmds-197 source, the zsh behavior from the zshparam man page, and the srand rule from POSIX. The GitHub examples come from two gh api search/issues queries ("shuf: command not found", 38 results, and "sort -R" with shuf, 114), and I read the quoted ones in full. The August sentence this follows up is in the timeout post.