Linux: xargs vs. exec {}
By Daniel Miessler on October 29th, 2006: Tagged as Linux | Sysadmin | Unix
There is a bit of a debate in some circles about using xargs vs. the -exec {} option that’s built into find itself. To me, however, it’s not much of a debate; -exec isn’t nearly as good as xargs for what I use find for. I tend to use it to perform tasks involving many files. “Move all these files there”, “copy all those directories there”, “Delete these links.”, etc.
This is where-execbreaks down andxargsshows its superiority. When you use-execto do the work you run a separate instance of the called program for each element of input. So iffindcomes up with 10,000 results, you runexec10,000 times. Withxargs, you build up the input into bundles and run them through the command as few times as possible, which is often just once. When dealing with hundreds or thousands of elements this is a big win forxargs.
That’s all nice and stuff, but you probably want to see it in action, right? Let’s run some numbers. Below is a listing of 1,668 .jpg files on my OS X system using both -exec and xargs:
# time find . -name "*.jpg" -exec ls {} \;
real 0m6.618s user 0m1.465s sys 0m4.396sHmm, that’s not bad — seven seconds for over around 1,600 files, right? Let’s try it with
xargs.
# time find . -name "*.jpg" -print0 | xargs -0 ls
real 0m1.120s user 0m0.594s sys 0m0.527sThat's one (1) second vs seven (7) seconds. Seriously;
xargs is the way to go.
Linux: Harnessing The Über-Powerful Find Command (+xargs)
By Daniel Miessler on October 10th, 2006: Tagged as Linux | Sysadmin
I just posted a new /study article about how to leverage the power of find and xargs:
Adding Ads
By Daniel Miessler on October 2nd, 2006: Tagged as Blogging | Business | Sysadmin
So part of my big move is going to be putting ads back on the site. Hopefully this won’t get much pushback; it’s really little more than an attempt to recoup the additional cost of hosting the site under the new model.
I also think it adds to the perceived legitimacy of a site to have ads on it. I know that seems kind of weird, but when you go to a site that has a bunch of ads on it you usually think, “this site must be pretty serious.”
Anyway, hopefully nobody will mind the additions; they’re pretty minimal really, with just a column going along the right side in both the blog area and the main content. If you do happen to be annoyed by it then let me know.
Getting Into Ruby
By Daniel Miessler on September 23rd, 2006: Tagged as Programming | Python | Ruby | Sysadmin
I feel like I’m the last person in the world to do this. Up until now I’ve pretty much chosen Python as my “Perlesque” language of choice, and I never had much of a will to dabble in what seemed to be nothing more than a slightly better Python.
But I changed my mind.
After spending some time at this live demo site, I decided I’m going to do my next little coding project in Ruby. I’m going to get the O’Reilly book on the language, actually, and learn it the right way. I might even consider coding my mst tool in this instead of Python. All depends on how Ruby interracts with OS commands, really.
I’m excited, yet I feel a bit like I’ve cheated on Python. We’ll see how it goes…
Site Updates: New Server, New Hosting Model
By Daniel Miessler on September 23rd, 2006: Tagged as Blogging | Geek | Social | Sysadmin
I’m moving from my current host soon and going to a new setup. Rather than hosting with one of the standard companies and leasing one of their dedicated systems, I’m getting my own server and colo’ing it at a company administered by a friend of mine.
This is the final stage in my website’s evolution, and while it’s going to be somewhat expensive I’m going to try and offset some of the cost by putting ads back on the site. I’m hoping I can, without any effort, reach the $100 cuttoff for Google Ads and get that money to put toward the cost of the solution.
Plus, having ads on the site actually lends to credibility I’ve found. People subconsciously associate a strip of ads with the site being “professional”. Strange but true.
The Server
- CPU(s): Two Dual-Core Zeons at 1.6Ghz, 4MB Cache
- Memory: 4GB
- Drives: Two 36GB, 15,000rpm SAS Drives in RAID1 (PERC Controller)
That’s four (4) processor cores — each with a gig of RAM. I’m giddy about it, and I can’t wait to get it online with all my content swapped over. It was cheap as hell, too. I priced the same thing through IBM and it was out of control. Plus, the monthly payment model is good considering that’s how you take in ad revenue.
The main goal of the move is to be able to survive the Digg Effect so that every month or so I can get slammed with traffic and avoid seeing comments like, “The site’s already down.” That’s just embarrassing.
I am also going to step up my efforts at content creation and organization. I have dozens of ideas for good study articles just waiting to be written and published. My Unix/Linux commands section is especially bare and I have a few commands that I can’t wait to write about.
New Approach
Overall I just want to refocus my efforts on my /study page. I’m not so much interested in the /writing area right now. I want to stay centered around creating high quality content that summarizes complex material. It’s the approach that benefits me the most because I end up farily versed in the stuff that interests me.
It’s also the most rewarding because people seem to enjoy that content the most, and they voice this approval either through private emails or through the social bookmarking sites.
Anyway, change is on the way. I’m hoping to see all aspects of the site’s quality head skyward.
Under Fire
By Daniel Miessler on September 14th, 2006: Tagged as Digg | LAMP | Social | Sysadmin
I just got pwned by the Digg effect — for like 15 hours — and I’m still unsure of where the bottleneck was.
Was it MYSQL? Was it Apache? I need to do some research. If it’s my host I’m going to pummel them, and if it’s inefficient code on my server, then I’m going to rectify that. I do know that I could still ping my site while the website was down. I could still SSH to it as well, and get mail.
I’m thinking it has to be AMP based. So embarrassing…I pay good money for a dedicated server, and I should to be able to withstand a Digg effect better than this. This can’t happen again.
If anyone’s a guru on optimizing servers to withstand traffic onslaughts, let me know.
My Guide To lsof
By Daniel Miessler on September 13th, 2006: Tagged as Linux | Security | Sysadmin | Unix
lsof is one of the most powerful Linux/Unix tools out there, but it’s relatively unknown and unused. I’ve just completed a short guide to some of its most useful functions. If you’re into *nix at all and are not familiar with this incredible tool, go have a gander.
What Mail Server Do The Fortune 100 Companies Use?
By Daniel Miessler on August 18th, 2006: Tagged as Business | Email | Internet | Sysadmin
Ever wonder what mail servers the top U.S. companies run? I found myself wondering what the top universities used a while back, and I just recently got curious as to what the Fortune 100 companies used as well. The results were quite interesting; it’s always fun to see what some of these admins put in their mail banners:
220 plgmler2.imr.gm.com ESMTP Sendmail GM Secure; Thu, 17 Aug 2006 23:23:21 -0500 220 mail11.disney.com Postfix EGGS and Butter 220 IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted 220 Medco - Ready 220 mailgate2.safeway.com ESMTP ***** SMTP Ready ***** 220 mail1.cat.com ESMTP Microsoft Exchange Internet Mail Service 5.5.2657.72) ready 220 Northrop Grumman Corporation Ready 220 mms01bas.mms.us.syntegra.com (IntraStore TurboSendmail) ESMTP Service ready 220 [12.43.78.22] ESMTP 220 wamu.com Proxy Server 220 fdsmail01.fds.com Microsoft ESMTP MAIL Service, Version: 5.0.2195.6713 ready at Fri, 18 Aug 2006 00:31:58 -0400Quite a few of the banners warn anyone reading not to use the service for spam, although I have to wonder whether or not a spammer has ever changed their behavior due to a warning like this.
We also get to see some personality in the way the servers say goodbye (SMTP 221):
221 Bye 221 Until later [69.214.242.13] 221 Catch you laterAnyway, here are the complete results as well as the stuff I used to run the test:
Input
Output
The Complete Banners The Exit Messages
–
** Note: There’s a fine line between being curious about interesting service banners and portscanning. If you don’t know what that difference is then you probably shouldn’t be doing this at home. :)
Necessary Tools: ldp.exe
By Daniel Miessler on July 20th, 2006: Tagged as Infosec | Sysadmin
Microsoft’s LDAP browser. Very useful tool — especially in infosec since pretty much every security applicance out there can use AD authentication. Use this tool to get exact user DNs.