Linux: xargs vs. exec {}

By Daniel Miessler on October 29th, 2006: Tagged as Linux | Sysadmin | Unix

There is a bit of a debate in some circles about using xargs vs. the -exec {} option that’s built into find itself. To me, however, it’s not much of a debate; -exec isn’t nearly as good as xargs for what I use find for. I tend to use it to perform tasks involving many files. “Move all these files there”, “copy all those directories there”, “Delete these links.”, etc.

This is where -exec breaks down and xargs shows its superiority. When you use -exec to do the work you run a separate instance of the called program for each element of input. So if find comes up with 10,000 results, you run exec 10,000 times. With xargs, you build up the input into bundles and run them through the command as few times as possible, which is often just once. When dealing with hundreds or thousands of elements this is a big win for xargs.

That’s all nice and stuff, but you probably want to see it in action, right? Let’s run some numbers. Below is a listing of 1,668 .jpg files on my OS X system using both -exec and xargs:

# time find . -name "*.jpg" -exec ls {} \; 

real    0m6.618s
user    0m1.465s
sys     0m4.396s
Hmm, that’s not bad — seven seconds for over around 1,600 files, right? Let’s try it with xargs.

# time find . -name "*.jpg" -print0 | xargs -0 ls 

real    0m1.120s
user    0m0.594s
sys     0m0.527s
That's one (1) second vs seven (7) seconds. Seriously; xargs is the way to go.

  • Digg
  • Reddit
  • HackerNews
  • StumbleUpon
  • DZone
  • Google Bookmarks
  • Twitter
  • Facebook
  • del.icio.us
  • FriendFeed
  • Posterous
  • RSS
  • email
  

View Comments

  • dchen
    You can have the same behavior with find by replacing the ; with a +. To illustrate, try running the following:

    find . -exec echo {} \;
    find . -exec echo {} +
    find . -print0 | xargs -0 echo
  • Justin McBroom
    tomchuck,

    bash has no problem looping through a list of files with our without spaces, you just
    have to make sure and put quotes around the loop variable:

    for png in *.png ; do echo pngcrush "$png"; done
  • When using the -i or -I string replacement for xargs (needed for any
    complex operation such as the convert example above), then it
    becomes impossible for xargs to group the arguments together and
    it has to execute them one at a time as in -exec. If you change the
    example above to:

    # time find . -name “*.jpg” -print0 | xargs -0 -ifile ls file

    you'll see it has almost exactly the same running time as -exec.

    This dual behavior of xargs can be surprising - a complex command
    written with -i may work fine since it only receives one argument at
    a time, but break suddenly if the -i is removed and the command
    faces batched arguments. It's probably best to use the -exec
    option to find(1) wherever you would need to use -i in xargs to avoid
    this confusion.
  • tomchuk
    Mike,

    bash loops work great until you hit a filename with a space.
  • To those who mentioned caching, I performed the test at least ten times using both commands to produce the numbers I did. I ran into the caching issue as well on other systems, but on the system in question I think I was getting real numbers. Again, both numbers come from the command being run multiple times. I didn't run one without the cache and one with it.

    That, I agree, would have been lame...
  • Mike
    mikkom,

    If you're using 'bash' as your shell (you should be), it's often easier to just say 'for i in *.jpg; do convert file blah...
  • admp
    TD: it's filesystem kernel-level cache. This is simply incorrect method of benchmarking. May the author check inner details better next time. Kudos.
  • TD
    If you try the commands in the other order, i.e. xargs first then -exec the exec is faster, at least in my little test of finding all jpgs in my home dir. xargs took 2mins exec took 21 secs.

    /me wonders if find caches results or something to speed it up when you do a similar find afterwards.
  • What impant did harddrive caching have on your test?
  • mikkom:
    I'm not too familiar with xargs, but I think this find expression will do what you want:

    find . -maxdepth 1 -name '*jpg' -exec convert '{}' -resize 90x -crop 90×90 0 0 '{}' ';'

    I usually put '' around {} in case of spaces.
  • .. and if you are doing a hardcore batch job for 15000 files, are those 6 seconds of shell processing really that important?

    I use xargs but definitely for different reasons than few second difference... I'm not aware if the following example is even possible with -exec:

    ls *jpg|xargs -ifile convert file -resize 90x -crop 90x90 0 0 file
blog comments powered by Disqus

 

twitter_icon

Sample Original Content


Information Security

Tutorials and Primers

Culture & Society

Technology & Science

Politics

Philosophy & Religion

Miscellaneous

Tools & Projects


Blog Archives