Linux: xargs vs. exec {}

October 29th, 2006 | Linux | Sysadmin | Unix

There is a bit of a debate in some circles about using xargs vs. the -exec {} option that’s built into find itself. To me, however, it’s not much of a debate; -exec isn’t nearly as good as xargs for what I use find for. I tend to use it to perform tasks involving many files. “Move all these files there”, “copy all those directories there”, “Delete these links.”, etc.

This is where -exec breaks down and xargs shows its superiority. When you use -exec to do the work you run a separate instance of the called program for each element of input. So if find comes up with 10,000 results, you run exec 10,000 times. With xargs, you build up the input into bundles and run them through the command as few times as possible, which is often just once. When dealing with hundreds or thousands of elements this is a big win for xargs.

That’s all nice and stuff, but you probably want to see it in action, right? Let’s run some numbers. Below is a listing of 1,668 .jpg files on my OS X system using both -exec and xargs:

# time find . -name "*.jpg" -exec ls {} \; 

real    0m6.618s
user    0m1.465s
sys     0m4.396s
Hmm, that’s not bad — seven seconds for over around 1,600 files, right? Let’s try it with xargs.

# time find . -name "*.jpg" -print0 | xargs -0 ls 

real    0m1.120s
user    0m0.594s
sys     0m0.527s
That's one (1) second vs seven (7) seconds. Seriously; xargs is the way to go.

 
2 Enjoyed
Don't
move!


☐ If you enjoyed this, please consider connecting via Twitter, RSS, or my other content.

  • http://www.blosker.com/ mikkom

    .. and if you are doing a hardcore batch job for 15000 files, are those 6 seconds of shell processing really that important?

    I use xargs but definitely for different reasons than few second difference… I’m not aware if the following example is even possible with -exec:

    ls *jpg|xargs -ifile convert file -resize 90x -crop 90×90 0 0 file

  • http://www.blosker.com mikkom

    .. and if you are doing a hardcore batch job for 15000 files, are those 6 seconds of shell processing really that important?

    I use xargs but definitely for different reasons than few second difference… I’m not aware if the following example is even possible with -exec:

    ls *jpg|xargs -ifile convert file -resize 90x -crop 90×90 0 0 file

  • http://rory.netsoc.ucd.ie/blog Rory

    mikkom: I’m not too familiar with xargs, but I think this find expression will do what you want:

    find . -maxdepth 1 -name ‘*jpg’ -exec convert ‘{}’ -resize 90x -crop 90×90 0 0 ‘{}’ ‘;’

    I usually put ” around {} in case of spaces.

  • http://rory.netsoc.ucd.ie/blog Rory

    mikkom: I’m not too familiar with xargs, but I think this find expression will do what you want:

    find . -maxdepth 1 -name ‘*jpg’ -exec convert ‘{}’ -resize 90x -crop 90×90 0 0 ‘{}’ ‘;’

    I usually put ” around {} in case of spaces.

  • http://pieterbreed.blogspot.com/ Pieter Breed

    What impant did harddrive caching have on your test?

  • http://pieterbreed.blogspot.com/ Pieter Breed

    What impant did harddrive caching have on your test?

  • TD

    If you try the commands in the other order, i.e. xargs first then -exec the exec is faster, at least in my little test of finding all jpgs in my home dir. xargs took 2mins exec took 21 secs.

    /me wonders if find caches results or something to speed it up when you do a similar find afterwards.

  • TD

    If you try the commands in the other order, i.e. xargs first then -exec the exec is faster, at least in my little test of finding all jpgs in my home dir. xargs took 2mins exec took 21 secs.

    /me wonders if find caches results or something to speed it up when you do a similar find afterwards.

  • admp

    TD: it’s filesystem kernel-level cache. This is simply incorrect method of benchmarking. May the author check inner details better next time. Kudos.

  • admp

    TD: it’s filesystem kernel-level cache. This is simply incorrect method of benchmarking. May the author check inner details better next time. Kudos.

  • Mike

    mikkom,

    If you’re using ‘bash’ as your shell (you should be), it’s often easier to just say ‘for i in *.jpg; do convert file blah…

  • Mike

    mikkom,

    If you’re using ‘bash’ as your shell (you should be), it’s often easier to just say ‘for i in *.jpg; do convert file blah…

  • http://dmiessler.com/ Daniel Miessler

    To those who mentioned caching, I performed the test at least ten times using both commands to produce the numbers I did. I ran into the caching issue as well on other systems, but on the system in question I think I was getting real numbers. Again, both numbers come from the command being run multiple times. I didn’t run one without the cache and one with it.

    That, I agree, would have been lame…

  • http://dmiessler.com Daniel Miessler

    To those who mentioned caching, I performed the test at least ten times using both commands to produce the numbers I did. I ran into the caching issue as well on other systems, but on the system in question I think I was getting real numbers. Again, both numbers come from the command being run multiple times. I didn’t run one without the cache and one with it.

    That, I agree, would have been lame…

  • tomchuk

    Mike,

    bash loops work great until you hit a filename with a space.

  • tomchuk

    Mike,

    bash loops work great until you hit a filename with a space.

  • http://synthcode.com/ foof

    When using the -i or -I string replacement for xargs (needed for any complex operation such as the convert example above), then it becomes impossible for xargs to group the arguments together and it has to execute them one at a time as in -exec. If you change the example above to:

    time find . -name “*.jpg” -print0 | xargs -0 -ifile ls file

    you’ll see it has almost exactly the same running time as -exec.

    This dual behavior of xargs can be surprising – a complex command written with -i may work fine since it only receives one argument at a time, but break suddenly if the -i is removed and the command faces batched arguments. It’s probably best to use the -exec option to find(1) wherever you would need to use -i in xargs to avoid this confusion.

  • http://synthcode.com/ foof

    When using the -i or -I string replacement for xargs (needed for any complex operation such as the convert example above), then it becomes impossible for xargs to group the arguments together and it has to execute them one at a time as in -exec. If you change the example above to:

    time find . -name “*.jpg” -print0 | xargs -0 -ifile ls file

    you’ll see it has almost exactly the same running time as -exec.

    This dual behavior of xargs can be surprising – a complex command written with -i may work fine since it only receives one argument at a time, but break suddenly if the -i is removed and the command faces batched arguments. It’s probably best to use the -exec option to find(1) wherever you would need to use -i in xargs to avoid this confusion.

  • Justin McBroom

    tomchuck,

    bash has no problem looping through a list of files with our without spaces, you just have to make sure and put quotes around the loop variable:

    for png in *.png ; do echo pngcrush “$png”; done

  • Justin McBroom

    tomchuck,

    bash has no problem looping through a list of files with our without spaces, you just have to make sure and put quotes around the loop variable:

    for png in *.png ; do echo pngcrush “$png”; done

  • neuralstatic

    that does not match my experience, justin.
    even if it did echo it as a complete name (which it doesn't for me), then acting on that variable is still delimited at the first space, right?

  • dj

    Doing some reading on find, xargs and exec. I don't know when it was added, but on the -exec, change the “;” to “+” to “aggregated into sets”. The test results will change a lot.
    See: http://www.opengroup.org/onlinepubs/009695399/u…

    >”variable is still delimited at the first space”
    Check out the documentation at: http://www.tldp.org (Short for: The Linux Documentation Project) on bash, quotes, and file name expansion.

  • dj

    Doing some reading on find, xargs and exec. I don't know when it was added, but on the -exec, change the “;” to “+” to “aggregated into sets”. The test results will change a lot.
    See: http://www.opengroup.org/onlinepubs/009695399/u…

    >”variable is still delimited at the first space”
    Check out the documentation at: http://www.tldp.org (Short for: The Linux Documentation Project) on bash, quotes, and file name expansion.

  • dchen

    You can have the same behavior with find by replacing the ; with a +. To illustrate, try running the following:

    find . -exec echo {} ;
    find . -exec echo {} +
    find . -print0 | xargs -0 echo

  • dchen

    You can have the same behavior with find by replacing the ; with a +. To illustrate, try running the following:

    find . -exec echo {} ;
    find . -exec echo {} +
    find . -print0 | xargs -0 echo

  • Pingback: http://takescreenshotonmac.com

  • Pingback: here

  • Pingback: Pharr

  • Pingback: Buildasign Coupon

  • Pingback: LSAT logic games

  • Pingback: other

  • Pingback: Apartments Vrtlici

  • Pingback: car insurance quote

  • Pingback: โชคดี

  • Pingback: krtasiye

  • Pingback: messebau I standbau I prosoektstaender I prospektstander I veranstaltungstechnik I plakatständer I plexidisplay I klapprahmen I infostander I infostaender I stehtische I veranstaltungstechnik I messestand I roll-ups I werbdedisplay I Sicherheitsspiegel I