Linux: `xargs` vs. `exec {}`

By Daniel Miessler on October 29th, 2006: Tagged as Linux | Sysadmin | Unix

There is a bit of a debate in some circles about using xargs vs. the -exec {} option that’s built into find itself. To me, however, it’s not much of a debate; -exec isn’t nearly as good as xargs for what I use find for. I tend to use it to perform tasks involving many files. “Move all these files there”, “copy all those directories there”, “Delete these links.”, etc.

This is where -exec breaks down and xargs shows its superiority. When you use -exec to do the work you run a separate instance of the called program for each element of input. So if find comes up with 10,000 results, you run exec 10,000 times. With xargs, you build up the input into bundles and run them through the command as few times as possible, which is often just once. When dealing with hundreds or thousands of elements this is a big win for xargs.

That’s all nice and stuff, but you probably want to see it in action, right? Let’s run some numbers. Below is a listing of 1,668 .jpg files on my OS X system using both -exec and xargs:

# time find . -name "*.jpg" -exec ls {} \;

real    0m6.618s
user    0m1.465s
sys     0m4.396s

Hmm, that’s not bad — seven seconds for over around 1,600 files, right? Let’s try it with xargs.

# time find . -name "*.jpg" -print0 | xargs -0 ls

real    0m1.120s
user    0m0.594s
sys     0m0.527s


That's one (1) second vs seven (7) seconds. Seriously; xargs is the way to go.
Related Content
Linux: Harnessing The Über-Powerful Find Command (+xargs)
Your Problem with Vim is That You Don’t Grok Vi
Ubuntu: aptitude > apt-*
Getting with Git
Learning git









http://www.blosker.com/ mikkom



.. and if you are doing a hardcore batch job for 15000 files, are those 6 seconds of shell processing really that important?
I use xargs but definitely for different reasons than few second difference… I’m not aware if the following example is even possible with -exec:
ls *jpg|xargs -ifile convert file -resize 90x -crop 90×90 0 0 file






http://www.blosker.com mikkom



.. and if you are doing a hardcore batch job for 15000 files, are those 6 seconds of shell processing really that important?
I use xargs but definitely for different reasons than few second difference… I’m not aware if the following example is even possible with -exec:
ls *jpg|xargs -ifile convert file -resize 90x -crop 90×90 0 0 file






http://rory.netsoc.ucd.ie/blog Rory



mikkom:
I’m not too familiar with xargs, but I think this find expression will do what you want:
find . -maxdepth 1 -name ‘*jpg’ -exec convert ‘{}’ -resize 90x -crop 90×90 0 0 ‘{}’ ‘;’
I usually put ” around {} in case of spaces.






http://rory.netsoc.ucd.ie/blog Rory



mikkom:
I’m not too familiar with xargs, but I think this find expression will do what you want:
find . -maxdepth 1 -name ‘*jpg’ -exec convert ‘{}’ -resize 90x -crop 90×90 0 0 ‘{}’ ‘;’
I usually put ” around {} in case of spaces.






http://pieterbreed.blogspot.com/ Pieter Breed



What impant did harddrive caching have on your test?






http://pieterbreed.blogspot.com/ Pieter Breed



What impant did harddrive caching have on your test?






TD



If you try the commands in the other order, i.e. xargs first then -exec the exec is faster, at least in my little test of finding all jpgs in my home dir. xargs took 2mins exec took 21 secs.
/me wonders if find caches results or something to speed it up when you do a similar find afterwards.






TD



If you try the commands in the other order, i.e. xargs first then -exec the exec is faster, at least in my little test of finding all jpgs in my home dir. xargs took 2mins exec took 21 secs.
/me wonders if find caches results or something to speed it up when you do a similar find afterwards.






admp



TD: it’s filesystem kernel-level cache. This is simply incorrect method of benchmarking. May the author check inner details better next time. Kudos.






admp



TD: it’s filesystem kernel-level cache. This is simply incorrect method of benchmarking. May the author check inner details better next time. Kudos.






Mike



mikkom,
If you’re using ‘bash’ as your shell (you should be), it’s often easier to just say ‘for i in *.jpg; do convert file blah…






Mike



mikkom,
If you’re using ‘bash’ as your shell (you should be), it’s often easier to just say ‘for i in *.jpg; do convert file blah…






http://dmiessler.com/ Daniel Miessler



To those who mentioned caching, I performed the test at least ten times using both commands to produce the numbers I did. I ran into the caching issue as well on other systems, but on the system in question I think I was getting real numbers. Again, both numbers come from the command being run multiple times. I didn’t run one without the cache and one with it.
That, I agree, would have been lame…






http://dmiessler.com Daniel Miessler



To those who mentioned caching, I performed the test at least ten times using both commands to produce the numbers I did. I ran into the caching issue as well on other systems, but on the system in question I think I was getting real numbers. Again, both numbers come from the command being run multiple times. I didn’t run one without the cache and one with it.
That, I agree, would have been lame…






tomchuk



Mike,
bash loops work great until you hit a filename with a space.






tomchuk



Mike,
bash loops work great until you hit a filename with a space.






http://synthcode.com/ foof



When using the -i or -I string replacement for xargs (needed for any
complex operation such as the convert example above), then it
becomes impossible for xargs to group the arguments together and
it has to execute them one at a time as in -exec. If you change the
example above to:
time find . -name “*.jpg” -print0 | xargs -0 -ifile ls file
you’ll see it has almost exactly the same running time as -exec.
This dual behavior of xargs can be surprising – a complex command
written with -i may work fine since it only receives one argument at
a time, but break suddenly if the -i is removed and the command
faces batched arguments. It’s probably best to use the -exec
option to find(1) wherever you would need to use -i in xargs to avoid
this confusion.






http://synthcode.com/ foof



When using the -i or -I string replacement for xargs (needed for any
complex operation such as the convert example above), then it
becomes impossible for xargs to group the arguments together and
it has to execute them one at a time as in -exec. If you change the
example above to:
time find . -name “*.jpg” -print0 | xargs -0 -ifile ls file
you’ll see it has almost exactly the same running time as -exec.
This dual behavior of xargs can be surprising – a complex command
written with -i may work fine since it only receives one argument at
a time, but break suddenly if the -i is removed and the command
faces batched arguments. It’s probably best to use the -exec
option to find(1) wherever you would need to use -i in xargs to avoid
this confusion.






Justin McBroom



tomchuck,
bash has no problem looping through a list of files with our without spaces, you just
have to make sure and put quotes around the loop variable:
for png in *.png ; do echo pngcrush “$png”; done






Justin McBroom



tomchuck,
bash has no problem looping through a list of files with our without spaces, you just
have to make sure and put quotes around the loop variable:
for png in *.png ; do echo pngcrush “$png”; done






neuralstatic



that does not match my experience, justin.
even if it did echo it as a complete name (which it doesn't for me), then acting on that variable is still delimited at the first space, right?






dj



Doing some reading on find, xargs and exec. I don't know when it was added, but on the -exec, change the “;” to “+” to “aggregated into sets”. The test results will change a lot.
See: http://www.opengroup.org/onlinepubs/009695399/u…

>”variable is still delimited at the first space”
Check out the documentation at: http://www.tldp.org (Short for: The Linux Documentation Project) on bash, quotes, and file name expansion.






dj



Doing some reading on find, xargs and exec. I don't know when it was added, but on the -exec, change the “;” to “+” to “aggregated into sets”. The test results will change a lot.
See: http://www.opengroup.org/onlinepubs/009695399/u…

>”variable is still delimited at the first space”
Check out the documentation at: http://www.tldp.org (Short for: The Linux Documentation Project) on bash, quotes, and file name expansion.






dchen



You can have the same behavior with find by replacing the ; with a +. To illustrate, try running the following:

find . -exec echo {} ;
find . -exec echo {} +
find . -print0 | xargs -0 echo






dchen



You can have the same behavior with find by replacing the ; with a +. To illustrate, try running the following:

find . -exec echo {} ;
find . -exec echo {} +
find . -print0 | xargs -0 echo

Linux: `xargs` vs. `exec {}`

By Daniel Miessler on October 29th, 2006: Tagged as Linux | Sysadmin | Unix

Related Content

time find . -name “*.jpg” -print0 | xargs -0 -ifile ls file

Top

Popular

Information Security / Technology

Politics

Philosophy & Religion

Technology & Science

Culture & Society

Miscellaneous

Arguments

Projects

Collections

Twitter

What I'm Reading

Favorite Books and Essays

Top Blog Categories

Inputs

Linux: xargs vs. exec {}

By Daniel Miessler on October 29th, 2006: Tagged as Linux | Sysadmin | Unix

Related Content

time find . -name “*.jpg” -print0 | xargs -0 -ifile ls file

Top

Popular

Information Security / Technology

Politics

Philosophy & Religion

Technology & Science

Culture & Society

Miscellaneous

Arguments

Projects

Collections

Twitter

What I'm Reading

Favorite Books and Essays

Top Blog Categories

Inputs

Linux: `xargs` vs. `exec {}`