Cobbling & Evangelizing Technology

Windows vs. Mac OS X (It comes down to font rendering)

2011-12-25T22:57:00.000-08:00

When it comes to which operating system someone likes, it's very much based on what biases that person brings to the table. So no shock that people who have been on Windows for years have numerous complaints about Mac OS X or vice versa.

However, there are differences that have nothing to do with user interface design choices such as toolbars (or lack thereof), keyboard shortcuts, etc., etc. One of the biggest differences between the two platforms is fonts, but more specifically, font rendering. If you've used Windows for years, one of the first things you'll notice as you start using Mac OS X is that things, e.g., web pages, look different and indeed it's not just your imagination. It turns out how Apple renders fonts is different than how Microsoft does it on Windows:

http://www.joelonsoftware.com/items/2007/06/12.html

I became acutely aware of the latter back in 2007 on account of noticing that the Windows version of Apple's Safari browser made pages look different than what I had been accustomed to while browsing with Firefox and IE (before I stopped using it years prior).

Turns out, now that I've had a Mac for the past year, I've reached a point where I prefer browsing on my Macintosh vs. my Windows system because of this font rendering difference (how pages look).

While I would agree with Spolsky that Windows' fonts are easier to read, the difference isn't stunning. Text is clear on the Mac. It's just that Windows uses less anti-aliasing and the pixel contrast, i.e. the individual pixels that make up a single letter, is more pronounced on account of the jaggies. Spolsky also writes:

you'll find that most people don't really know what to choose, and will opt for the one that seems most familiar.

Iterating, after actively using Mac OS X for a year, I now prefer browsing on my Mac. It's created a bias that I would have to say, if I bought a laptop today, it would be an Apple Macbook. (Aside: The Mac I got 12 months ago is a Mac Mini).

While font rendering is subtle, since it is visual and vision is people's primary sense, it's a major anchor for ensconcing people into a comfort zone. And since people are wont to resist change, it's very hard to pry them away from said comfort zone once they gravitate to it. This all means that people dropping Windows in favor of Macintoshes aren't likely to come back anytime soon. As a bad portent for Microsoft, check out the following AppleInsider article that came out last month (Nov. 2011):

http://www.appleinsider.com/articles/11/11/14/mac_sales_surge_despite_slipping_european_pc_market.html

In summary, PC sales dropped double digits in both the UK & Germany and close to 10% in France. But if companies selling PCs in Europe are blaming the global recession, the Macintosh market for Apple in Western Europe grew just shy of 20% year over year. I'll also remind the reader that Apple charges a premium for its hardware.

In closing, while I very much still use Windows 7 on account of Windows Media Center and my XBox360 acting as my DVR (check out the latter video link), nowadays if I'm browsing the web, more than likely it's on an Apple device (my Mac, iPad 2 or iPhone).

No, Google's Chrome Browser isn't the fastest browser by leaps and bounds (anymore)

2011-09-27T21:08:00.000-07:00

Tech people (like the rest of the homosapiens) are experts at mixing facts with opinion, a.k.a. bias. In my view Firefox by and large caught up to Chrome with release 4.0. Since then it has continued to make great strides...

Yet, I've noticed many people saying "Firefox is bloated, slow, blah, blah, blah, blah, blah". One of them (who I don't know personally) is ZDNet blogger Steven J. Vaughan-Nichols. In short, Steven trashes Firefox with every major release but worships Chrome's every release. Merely juxtapose what he wrote not even one month ago about Firefox 6 when it was released (the Mozilla team this year adopted a fast release cycle like Google's for Chrome):

http://www.zdnet.com/blog/networking/firefox-6-a-firefox-too-far-review/1380

Now compare that with what he wrote about Chrome 14 which came out a few days ago:

http://www.zdnet.com/blog/networking/chrome-14-the-best-web-browser-keeps-getting-better-review/1469?tag=search-results-rivers;item3

There's opinion, "Firefox is bloated", then there's reality, Lifehacker just juxtaposed Firefox 7 with other major browsers:

http://lifehacker.com/5844150/browser-speed-tests-firefox-7-chrome-14-internet-explorer-9-and-more

Going back not even a month ago Tom's Hardware did JavaScript performance tests comparing the major browsers:

http://www.tomshardware.com/reviews/web-browser-performance-standard-html5,3013-9.html

In terms of JavaScript performance, Firefox 6 beat Chrome 13 in 4 of 5 tests. And today Firefox 7 was released. In closing, if your frame of reference of Firefox is anchored in the distant past, you should look again. Beyond that, if the plugins you've used under Firefox have memory leaks, crash Firefox, etc., etc., that's not the Mozilla team's fault.

I should also point out that Firefox for many years has supported HTTP Pipelining. Chrome to this day does not support HTTP Pipelining which is part of the HTTP 1.1 specification that came out 10+ years ago. Chrome does feature an alternative called SPDY however this is not part of the HTTP specification and you will only benefit from this if you are visiting Google web properties. The only popular browser that readily supports HTTP Pipelining is Firefox. One notable benefit is that as your connection latency goes up, e.g., 3G aircard, tethering on your laptop or God forbid, dialup, performance improves. If you find yourself browsing a lot on a 3G connection then Firefox with HTTP Pipelining is for you. Performance also goes up as the number of distinct elements that need to be fetched goes up, e.g., lots of images (which translate to that many more HTTP requests).

HTTP Pipelining is the first thing I enable when I have a new Firefox profile. In the address bar simply type "about:config", search for "pipe" then set network.http.pipelining to true. I also change network.http.pipelining.maxrequests to 7.

Microsoft's Enhanced Mitigation Experience Toolkit

2010-09-12T18:40:00.000-07:00

A week ago Microsoft released version 2.0 of EMET (Enhanced Mitigation Experience Toolkit):

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=c6f0a6ee-05ac-4eb6-acd0-362559fd2f04&displayLang=en

Don't know what EMET is? I highly suggest you use it to launch applications that talk on the Net, in particular your browser. Here's a very technical video from Microsoft that talks about EMET:

http://technet.microsoft.com/en-us/security/ff859539.aspx

Let me give you a sample scenario. You visit a legitimate site that you've used for ages which unbeknownst to you, ads being served up are coming from a compromised ad server (a scenario which by the way has happened many times). The malware then attempts to leverage an arbitrary code execution flaw. Unfortunately for you, you're not very diligent about keeping your system up to date or you've ignored updating your system because well, "I'll do it later." Malware sent your way succeeds in leveraging an arbitrary code execution flaw that just surfaced with your browser of choice two days ago installing a backdoor and thus gaining complete control of your computer at which point the remote attacker can take whatever files they please, use your computer as part of a spam network, denial of service network, etc, etc. In short, your system is completely at someone else's mercy and you don't even know it. Let's take a more optimistic scenario. You're on a fully patched Windows 7 system with UAC enabled so you're safe (usually) from getting your machine taken over but malware comes in through your browser which isn't patched. You don't have the latest browser revision because you've put it off, turned off auto-updates or worse, there's no patch for an exploit that has surfaced. You're then unfortunate enough to visit a site with malware and a recent exploit is leveraged introducing rogue code into your system. That code is at the very least capable of reading and modifying files you use day to day. Whether they be explicit documents (such as MS Word) or implicit documents (the cookies in your browser). Unfortunately, your browser doesn't prevent the malicious code from reading any file(s) belonging to you, in particular, browser cookies. After which, someone starts going into your various online accounts with your active cookies (which were conveniently sent to them over the Net) to see what they can find.

So how do you use EMET?

1) Go to the first link I provided - download and install EMET
2) After launching EMET hit the Configure Apps button in the lower right
3) Hit the Add button on the dialog box that comes up and specify the path to an executable you would like to protect, e.g.:

C:\Program Files (x86)\Mozilla Firefox\firefox.exe

4) Hit the Open button on the file browsing dialog (aka OK).
5) Restart the application in question, in this example, Firefox

(Look at blog post image)

Firefox is now protected from a variety of attack vectors often used in arbitrary code execution. The video elaborates on them quite well.

Whereas RemoveAdmin (a security tool that I authored) is all about leveraging OS level security, Microsoft's EMET is about maintaining the integrity of processes and thus, at the very least, providing application level security, e.g., your browser cookies. At worst, if you have an unpatched system (the OS) you could find yourself with a system that's been botted, has had a keyboard logger installed, etc., etc.

In my particular case, I not only have added the browsers I use day to day to EMET (Chrome, Firefox), I've added all applications I regularly use that talk on the Internet. In particular, iTunes, WinAmp, Outlook, Adobe's PDF reader, Windows' Media Player and Apple's QuickTime player. The links I've provided in the previous sentence point to security advisories for each of these applications they are not links to the products' respective web pages. If you have doubts about what I'm saying, just visits those links. Yes, as hard as it for a lay person to comprehend, you can have your system compromised by watching a video pulled off a web site. This is why you should start using EMET today. In short, I will never launch my browser from here on out without this tool.

Finally the following article surfaced after my initial blog post. Here's a scenario where an exploit of Adobe's PDF reader has surfaced, Adobe itself doesn't yet have a patch but through the use of EMET the exploit is short-circuited:

http://news.cnet.com/8301-1009_3-20016161-83.html?tag=topTechContentWrap;editorPicks

The Rise, Rise and RISE of Apple

2010-07-22T11:30:00.000-07:00

Is the news at this point surprising?

Apple's Profit Rises 78% on Appeal of New Devices

Apple's position is further cemented by one simple fact. The iPhone has fundamentally changed the way people operate, i.e. not only the things I can do on a day to day basis while I'm on the move but also when I'm sitting at home. In contrast neither Windows or Mac OS X have fundamentally altered how I've interacted with personal computers for years. While in some cases they do provide interesting back end services such as Windows' Media Center, Netflix's streaming service is the primary manner in which I consume video and I don't need a personal computer for that.

With Apple's latest firmware iOS 4 I leave Skype running on the iPhone in the background all the time. I can make and receive various voice calls and bypass AT&T completely. At the same time Pandora runs in the background streaming music based on previous feedback I've given it. This all the while relaxing on the couch and web surfing. If Pandora isn't doing it for me, I can fire up RemoteX and control WinAmp to start streaming music from various different sites that are in my default playlist. The music then starts over the Klipsch 5.1 speaker system on the PC. If the music streams aren't doing it for me either, I can fire Apple's remote app, search through my entire iTunes music library for a specific musician/song and start playing it:

http://www.apple.com/itunes/remote/

Then again maybe later in the day I decide that I'd rather watch some video content. I can fire up Boxee on the PC, display it on my HDTV and control the experience through my iPhone:

http://blog.boxee.tv/2009/03/15/boxee-iphone-remote-app-available-on-the-app-store/

(Aside: If you haven't tried the latest Boxee beta you're doing yourself a disservice, check it out)

You see it's about choice, choice and even more choice.

This is the fundamental difference between the Apple of old and the Microsoft of old. Many years (+15?) ago Bill Gates used to talk about "information at your fingertips" but his vision was very desktop centric and predated the rise of the World Wide Web which was spearheaded by forces not aligned with Gates' visions and Microsoft's fortes. We've reached a point where it's about consumption at your fingertips. Nowhere is this better showcased than what happened to Flipboard recently:

http://bits.blogs.nytimes.com/2010/08/04/flipboard-stumbles-in-its-first-days/

Excerpt:

Flipboard, which uses Amazon Web Services, has been doubling its server capacity each day. The company would not reveal how many people have signed up for the service, but said it is a good percentage of iPad owners, of which there are more than 3.3 million.

Personal computers have been as much about content creation as content consumption which is why a desktop centric approach placed on a small device in the palm of your hand doesn't work and helps explain the runaway success of the iPad.

One other fundamental change Apple has elicited from consumers is they've opened up their wallets. While the sales of software to corporations has been par for the course for decades, consumers became extremely frugal quite some time ago getting very used to the idea of FREE. For example, I can recall the days when you had to pay for anti-virus software for your home computer, there's no reason anyone should pay for anti-virus software today. No doubt this is one factor why CompUSA exited the brick & mortar retail space.

Specifically, the rise of micro-transactions and digital distribution has put an end to the walled garden of various middle men relegating them to the dust bin. With many applications either free or 99 cents, Apple's App Store has ushered a new age where people buy software with no significant concern for the money they spend or the value they derive. After all, if it's a lemon, the most they're out of is 99 cents. Yes, no more $9.95 and $19.95 software specials thank you very much.

As middle men's leverage disappears with new digital distribution, it allows people that actually create products to achieve independent wealth:

http://bits.blogs.nytimes.com/2010/06/25/doodle-jump-reaches-five-million-downloads/

The creators of Doodle Jump have sold millions of units. And the math is quite simple, app developers get 70% of the sale which means Doodle Jump's creators are now collectively millionaires and no surprise they quit their day jobs to focus on their new entertainment software business.

This is the kind of story that would never have happened 5 years ago if you were talking mobile applications. It's no surprise that because of stories like this along with the iPhone's market/mindshare, developers have flocked to Apple's platform. It also means that Microsoft, Google, Nokia and RIM (Blackberry) have their work cut out for them trying to establish similar mobile software ecosystems.

While Google has made lots of progress with its Android platform, I have to point out that all of the things I mentioned that I did at home with my iPhone (examples I gave earlier of controlling my PC) have absolutely no reliance on the telephone network (AT&T). Everything I talked about can be done with an iPod Touch or an iPad (via Wifi). Meaning the same software that works on the iPhone works on the latter Apple devices and such a dynamic or complementary ecosystem doesn't exist for other vendors. While there is movement in the Android space with respect to tablets, I'm talking about the here and now.

Which precisely leads into my key point. The reason why Windows has had a vise grip on the desktop is because of its ecosystem of choices. Whether its software drivers to drive an old laser printer to DVD creation software to photo imaging software, you name it, a turnkey solution for the consumer on Windows is probably there.

Switching to the mobile market, until the iPhone came along, there was no such equivalent ecosystem of choices. The telephone companies with their typical lack of vision did not make installing applications easy nor did they pioneer the smart phone category with a partner. Apple created the category and AT&T was willing to go along for the ride. With the introduction of the Apple App Store, seemingly overnight an explosion of choices for consumers became an ecosystem of choices. Which means the average consumer (jailbreakers, hackers, idealists, open source zealots need not apply) stands to lose more peripheral consumption capabilities than gain if they leave Apple's platform. Not being able to do many things you're used to becomes an extremely powerful dissuading force and people are very much creatures of habit further reducing the chances of them jumping ship.

As Microsoft, Google, RIM and Nokia will find, unseating such an ecosystem of choices is not easy. The peripheral advantages often outweigh enticements like "FREE". This is why OpenOffice hasn't suddenly supplanted Microsoft Office despite being "FREE". Or why LINUX desktops haven't supplanted Windows desktops despite, again, being "FREE".

It boils down to the ecosystem of choices and the side effect of having lots of choice (vs. few or no choices).

Microsoft Security Essentials

2010-06-20T12:12:00.000-07:00

This weekend I moved away from the freebie AVG as my antivirus software to Microsoft Security Essentials (MSE). This Lifehacker article clinched the deal for me:

http://lifehacker.com/5559102/microsoft-security-essentials-finds-unknown-malware-but-avoids-false-positives

However it was this Arstechnica article that elevated MSE enough in my mind and started me on the road to this decision. Basically a new attack vector was concocted in the wild that bypasses practically all antivirus software, except for MSE:

http://arstechnica.com/microsoft/news/2010/05/microsoft-mse-safe-from-windows-kernel-hook-attack.ars

Here is an article covering the technical details on this new attack vector:

http://arstechnica.com/security/news/2010/05/multicore-cpus-move-attack-from-theoretical-to-practical.ars

Given that Microsoft Security Essentials has better detection of malware than AVG (according to the information presented by Lifehacker, which I trust), fewer false positives, is not vulnerable to kernel hook attacks and last but not least is FREE, I was sold.

MSE is available for Windows XP, Windows Vista and Windows 7.

"Safari 5 tested: Chrome, Opera still have JavaScript edge"

2010-06-10T10:32:00.000-07:00

I was pretty sure it wasn't my imagination when I opined that Apple's just released Safari 5 felt significantly faster than Safari 4 on Windows (note the closing paragraph):

http://arstechnica.com/apple/news/2010/06/safari-5-tested-chrome-opera-still-have-javascript-edge.ars

Make note of the Sunspider graph, namely IE 8's performance (or lack thereof). IE 9 however seems like it might be worthwhile to use. Yes, necessity or rather, loss of market share in Microsoft's case, is the mother of invention (swift kick in the pants). Just look at the trend for IE on this chart:

http://en.wikipedia.org/wiki/File:Usage_share_of_web_browsers_%28Source_Net_Applications%29.svg

Apple Releases Safari 5 Today

2010-06-07T18:34:00.001-07:00

As part of all the noise at the World Wide Developer's Conference today, Apple released the next major version (5.x) of their Safari web browser for the Macintosh and Windows platforms. Safari 5 is now available from Apple's web site:

http://www.apple.com/safari/download/

After some initial empirical testing, Safari 5 (on the PC) feels much faster than Safari 4. It seems Apple made a real attempt at polishing the Windows version of Safari whereas in the past it seemed Apple was releasing a tool so web developers & testers could see how things might look on Mac OS X (the premise being it wasn't readily available). It feels snappy as I write this entry on Blogger.com.

Here's Lifehacker's writeup on Safari 5:

http://lifehacker.com/5557731/first-look-at-whats-new-in-safari-5

Finally, the question of whether to make Safari or any other particular browser my default is irrelevant for me since I use the most excellent Browser Chooser:

http://lifehacker.com/5352193/browser-chooser-asks-you-which-browser-to-open-new-links-in

Pioneer's Kuro Elite: Best flat-panel HDTV ever

2010-06-02T08:17:00.001-07:00

The subject of this blog post is a quote from CNET:

http://news.cnet.com/8301-17938_105-10054347-1.html

A couple of weeks ago my Sony CRT HDTV died. The particular model I had it turns out was extremely prone to failure. Visiting forums, I found them littered with people with issues. In some cases within two or three years after the TV was purchased in the early 2000's. As I did my research I weighed the pros and cons of trying to get the damn thing fixed. It turns an electronics repair place happened to be literally one block away from where I lived and they were well acquainted with the Sony model and its failure problems (a bad sign). They told me it would likely be $350 if it was fixable.

The "if" caught my attention and started me on the path of moving to a flat panel. It was a difficult decision for me since the video fidelity of the CRT was unmatched by flat panels. Even so, based on the reading I did on various forums the CRT could be a lost cause and I didn't want to sink any money on a 250 lb. dead weight.

So two days after its death I visited both Fry's Electronics and Best Buy and I was not impressed with the flat panel HDTVs I saw. Then on the same day on a whim I decided to visit Magnolia Hi-Fi in Seattle. When a salesman approached me I immediately commented that all the HDTVs I had seen that day were mediocre and that I was there hoping to be wowed by an HDTV but that I wasn't holding my breath. Before not too long he asked, "Are you familiar with Pioneer's Elite line?" I nodded. He then said something that immediately raised an eyebrow, "Are you familiar with the Kuro line of displays?" I responded with an affirmative but I also knew Pioneer was no longer making the Kuro line and that that even used they were expensive. Pioneer's Kuro line were reference displays and they blew everything else out of the water when it came to 1080p video fidelity. But as my luck would have it, Magnolia's manager actively tries to find Kuros for 'finicky' people like myself (aka videophiles). I had already seen BluRay playback over HDMI on a 50" Kuro in early 2009 and was utterly and thoroughly impressed (up until then no flat panel had impressed me).

I didn't need to be sold on the Kuro line. I didn't act back then because my Sony CRT was still working and I was extremely content with it given it didn't exhibit the graininess of flat panels when playing non-HD content (there's still a lot of non-HD content out there, e.g., streaming Hulu). I had been hoping the CRT would last me until 2015 when perhaps 50" OLED flat panels were available en masse but that wasn't meant to be.

When the salesman told me they had a brand new 50" Kuro in house, there was nothing else for him to say, I laid down the plastic. Magnolia didn't have a stand for it in stock so I picked it up two weeks later (back ordered). As I made small talk with my sales person I asked him if he still had the other Kuro in stock (they had 2 when I picked mine up) and he said, "I'm not sure but now you have me curious." He dug around in his inventory system and not only was the other one gone but there were 20 individuals who had laid down money at Magnolia stores across the US to get their hands on a Kuro when and if they became available.

The other small detail is that the CNET review is for the Kuro with the TV tuner, the Kuro I purchased doesn't have a built in tuner:

http://www.pioneerelectronics.com/PUSA/Products/HomeEntertainment/PlasmaTVs+Monitors/EliteMonitors/ci.PRO-101FD.Kuro

I'm using the Samsung HD tuner I had been using on the Sony CRT. I pointed out the CNET review since they're a third party singing the praises of the Kuro. It's not uncommon for CNET to make a comparison to the Kuro displays when reviewing a higher end HDTV. However, up to this day, no one has surpassed the Kuro line.

All I can say is, I lucked out and I'm extremely pleased with the Kuro.

A Tale of Two iPads

2010-04-02T00:10:00.001-07:00

This NYTimes review sums up the two sentiments that are behind the iPad.

http://www.nytimes.com/2010/04/01/technology/personaltech/01pogue.html?ref=technology

In short, many tech people convince themselves of the lack of the iPad's utility due to comparisons with traditional computers. I might point out that some people privy to the early Internet believed it should not be made readily available to the masses. The reasons were varied but I assure you, the root of some of it was mere technical snobbery.

I completely agree with the New York Times reviewer's assessment, if you are not a tech person, Apple has created the computer for the rest of us. Funny thing is, I decided to do some googling to try to find references to that old Apple slogan but instead I found someone saying exactly what I'm saying here:

http://www.theroot.com/views/techies-are-wrong-about-ipad

The iPad is not even out yet (2 more days) and when you view the applications that are being released for it, it is truly impressive (the Netflix one took me by surprise):

http://gizmodo.com/5507569/gizmodos-essential-ipad-apps

I ask you one simple question - when was the last time you saw such a movement for any other device?

Ha! We beat Apple! (Dream on)

2010-03-21T10:34:00.000-07:00

The nay sayers lined up when Apple announced the iPad but arguments such as "I'll get a general tablet so I can run whatever I want" miss the point. The point being these same critics aren't running off to buy a competing device like the HP slate gizmo announced roughly two months prior. In short, the Apple iPad isn't intended for them (barring hot air).

I read an excellent article on Tom'sHardware entitled 5 Reasons Tablets Suck And You Won't Buy One

Point number 4 is key. It should have been number 1, underlined and accentuated with a headline tag <H>. Interface is why previous tablet offerings have been relegated to niche status. Is there a particular reason for this? Yes, Rob Enderle sums it up nicely:

http://technologizer.com/2010/03/08/future-windows/3/

Excerpt:

Now the most successful company as measured by profitability is Apple–who appears to be still locked into the proprietary hardware and software model that proceeded Windows’ success.

The problem with the PC model and the even more complex cell phone OS model was that no one person owned the customer experience. As a result, with the exception of Apple, PC vendors started doing stupid things like not assuring the service experience or putting software that reduced reliability (crapware) on the systems they were selling. In addition, complexity and an excessive focus on cost reductions got out of hand, significantly reducing the perceived quality of the system.

When the iPad's sales numbers are announced in a year (with unit sales in the millions), the usual will happen. Microsoft will feel the need to duplicate Apple's success and will assemble a crack team to provide a competitive offering. Haven't we been here before? Yes. Zune trying to take on the iPod. This after Microsoft's first response to the iPod Plays for Sure completely fell on its face.

Do you see a pattern here? Continual reaction to Apple's market success stories. This all underscores my strong opinion that Microsoft's leadership lacks vision. Steve Jobs despite his warts (he's not the nicest person if rumors are true) is a tech visionary.

Up until now, Microsoft has simply told hardware vendors to slap plain jane Windows on a keyboardless computer. Once again, complacency and myopia hold Microsoft back.

How Microsoft Lost the Platform War

2010-02-26T22:32:00.000-08:00

I'm a big fan of Joel Spolsky's blog:

http://joelonsoftware.com/

Joel Spolsky has a lot of software development experience. While having worked at Microsoft in the 90's it is clear from his writing that his talents are mutually exclusive from those days. Microsoft was just another employer on the road of life. Today he runs FogCreek Software in New York City. One of my favorite postings from Joel was entitled How Microsoft Lost the API War written back on June 13, 2004. In technological terms this is now ancient history. However, what Joel wrote back then is still relevant in terms of how Microsoft lost the hearts and minds of software developers. While writing that column Joel took a tangent:

Why Apple and Sun Can't Sell Computers
Well, of course, that's a little bit silly: of course Apple and Sun can sell computers, but not to the two most lucrative markets for computers, namely, the corporate desktop and the home computer. Apple is still down there in the very low single digits of market share and the only people with Suns on their desktops are at Sun. (Please understand that I'm talking about large trends here, and therefore when I say things like "nobody" I really mean "fewer than 10,000,000 people," and so on and so forth.)

I wrote to Joel Spolsky in the past year and mused how those words no longer hold true (about Apple anyway). I pointed out that half of my team members at work were operating with Apple Macbooks.

Then today I caught this on the New York Times:

OS X Share Up 29% in Past Year, Slowly Chipping Away at Microsoft

Excerpt:

Apple’s relative share has grown by 29.4% in the past year, while Windows lost 3.8%. Mobile increased the most in the past year, more than doubling its share of web consumption.

Adding fuel to the fire is another potential paradigm shift - tablet/slate computing. While there will be various Windows 7 based tablet products on the market, I have a very high degree of confidence Apple's sales of the iPad will likely marginalize these offerings. This is an opinion I immediately formed when the iPad was announced and it is write ups like that of Brian X. Chen from Wired that reinforce this, i.e. he didn't write "What the HP slate Means for the Future of Computing."

What the iPad Means for the Future of Computing

At this point you're guessing I'm writing this on a Macintosh. Nope. I use Windows 7 day to day. However, I'm not an ostrich with its head buried in the sand. I see reality. Microsoft ZDNet blogger Ed Bott seems to be on the same wavelength:

Like I said, apps matter.

So why is Microsoft incapable of responding appropriately? Read the words of a former Microsoft Vice President:

Microsoft’s Creative Destruction

In short, Microsoft is severely hobbled by a Balkinzation stemming from differing agendas among its product groups, a lack of vision (Ballmer at best is a Chief Operating Officer, he has no vision whatsoever and the longer he stays the more Microsoft is damaged) and the usual myopia found at a large company wanting to protect cash cows originating in the distant past (Office, Windows).

Until Microsoft has a come to Jesus moment (often precipitated by lots of layoffs & loss of market share) nothing will change. Except by then, it could very well be too late...

Adobe's Flash R.I.P.?

2010-02-04T02:37:00.000-08:00

Adobe is frustrated at Steve Jobs as Apple's mobile devices increasingly become a bigger player in the Internet experience. The iPhone platform which was released three years ago still does not have Flash and that does not seem like it will change anytime soon. But it's not entirely Apple's fault. Adobe's Flash platform isn't conducive to long battery usage. Simple as that.

The recently announced Apple iPad along with advances in HTML5 will only accelerate the trend of companies rebuking Adobe's Flash platform over the long term.

It turns out, Adobe's frustrations are well grounded (in fear). If content creators start considering the growing Apple userbase and stop leveraging Flash because they'd rather not alienate Apple device users, it means a long and slow death spiral for Adobe's platform. Fact: Adobe makes a LOT of money selling content creation software, Adobe doesn't make any money on people downloading the Flash plugin. Adobe is one of the largest software vendors for the Windows platform outside of Microsoft itself, e.g., Photoshop, Dreamweaver, PDF software, etc., etc. In short, Adobe makes a large portion of its revenue selling content creation tools. If one of its major formats dies, i.e. Flash, that's a serious blow to Adobe.

Case in point, remember when Real Networks blazed trails with respect to streaming audio/video over the Internet? Well, look where they are now. I can't remember the last time I installed any of Real's software and given the fact that practically no one puts out content in Real's format anymore, that won't be changing anytime soon.

I wrote about HTML5 a few days ago which shimmys itself in a position that essentially starts loosening Flash's vice grip on video playback. It seems I'm not the only one with this opinion:

"HTML vs. Flash: Can a turf war be avoided?"

http://news.cnet.com/8301-30685_3-20000037-264.html?tag=newsLeadStoriesArea.1

Here's an excerpt:

Bruce Lawson, Web standards evangelist for browser maker Opera Software, believes HTML and the other technologies inevitably will replace Flash and already collectively are "very close" to reproducing today's Flash abilities.

"The Web (including video, games, animation) is too vital a platform for business, communication, and society to be in the hands of any single vendor," Lawson said. "But it'll be a while; there is a huge body of existing content that uses Flash."

Both Lawson and I have come to the same conclusion - the long term future for Adobe's Flash platform is not bright (pun intended).

The (Internet) World In Two Hands

2010-01-28T00:15:00.000-08:00

David Morgenstern a blogger from ZDNet cuts to the heart of the matter as far as what Apple's newly iPad announced represents, "The iPhone is a device that puts the world into one hand; the iPad, two hands." Read on if you like:

http://blogs.zdnet.com/Apple/?p=5890&tag=wrapper;col1

I could not agree more. A consistent behavior I've seen of naysayers is a continual comparison of the newly announced iPad vs. laptops.

Here's one scenario that I don't see any laptop or netbook taking on. Increasingly HDTVs are coming with Ethernet ports to stream content off the Net. But who says they have to act purely as clients? That is, reading information, e.g., movies off Netflix's video servers. Why can't the TV be the server? Albeit a really simple server. Like the kind where instead of relying on infrared signals it abides by a well known command protocol coming through its Ethernet connection.

So before too long the TV industry devises a standard (that would be layer 7 application protocol for all the geeks) for controlling televisions via Ethernet interfaces. Then, before you know it, you'll see a remote control application for the iPad (or its posterity) where every living room device that has a remote will have its buttons represented as a skin overlaid on the iPad's surface. This is one of many scenarios that comes to mind where a laptop or netbook simply does not make the grade and looking at the iPad strictly through such a lens falls short.

Which is why I couldn't agree more with the title of Morgenstern's blog post "Dude, the iPad isn't a notebook - get over it!"

If you have doubts about the enthusiasm the iPad generated, I can tell you empirically that as Steve Jobs unveiled Apple's latest creation, tech sites were being hammered with traffic. I found Engadget and ZDNet to be noticeably slow as the morning of the iPad's unveiling wore on. I was following the announcement through:

http://live.gdgt.com/

I visited other tech sites since I was curious to see the buzz elsewhere - I found them to be very slow.

Chrome 4.0 / HTML 5 Video Playback

2010-01-26T23:58:00.000-08:00

Google's just released the next major revision of its Chrome browser, version 4.0, to the masses in the last day:

http://lifehacker.com/5456369/chromes-new-stable-release-brings-extensions-bookmark-sync-to-the-masses?autoplay=true

Previously Chrome 4.0 was only available to the initiated (development release). Updating to 4.0 for existing users is simply a matter of clicking on the wrench icon in the rightmost toolbar area and then selecting About Google Chrome.

DownloadSquad featured a variety of Chrome plugins in a November article (only useful at the time to folks with the development release):

http://www.downloadsquad.com/2009/11/02/15-great-google-chrome-extensions/

With 4.0's release Chrome's extensibility goes up dramatically and thus its utility.

HTML 5

Chrome's addition of extensions overshadow its in HTML 5. Among them, video playback. The ramifications of this while not immediately forthcoming are significant. For quite some time most web designers/developers have assumed their users to have Flash but it turns out Flash has its issues. For starters, Flash's performance under Mac OS X is very poor and something that many Mac fans continually scoff at. The proof is in the pudding:

http://www.engadget.com/2009/10/29/27-inch-imacs-having-performance-issues/

For a variety of reasons including the fact that the HyperText Markup Language (HTML) originally was focusing on document publication and not rich multimedia experiences, Macromedia (a well known company later acquired by Adobe) entrenched itself with the novel idea of extending browsers with video playback. Before you knew it, its Flash platform became the defacto standard for video playback. The fly in the ointment is that as the Macintosh user base has grown dramatically, they're increasingly a louder voice and unhappy with a second rate video playback platform. Rest assured, the number of Macintosh users is growing faster than Adobe (and Steve Ballmer) would like - Apple's recently announced latest quarterly results were phenomenal.

There are two big issues with Flash. First, Adobe has struggled with lots of security issues with both its Flash plugin and PDF reader. Secondly, as much as I've been a fan of Flash for blazing trails, I hate to say this but I'm liking the idea of not requiring it on my desktop. It turns out performance-wise, by today's standards its architecture poorly leverages my hardware. Despite Flash being pervasive for the better part of a decade Adobe only now has seen it fit to leverage GPUs during video playback. GPU acceleration within Flash is still in beta:

http://www.engadget.com/2009/11/17/adobes-flash-player-10-1-beta-gpu-acceleration-tested-document/

In other words, Flash can be very CPU intensive while your computer' graphics subsystem and its computational prowess goes unused. It turns out decoding video is intrinsically a parallel computation, something GPUs excel at.

I'm willing to bet anyone with a laptop, even a powerful one, knows all too well what happens when you start playing lots of Flash content - the fan starts cranking.

I'll close with pointing out that if you are running Chrome 4.0, you can opt to use HTML 5 video playback by default on YouTube instead of Flash by visiting the following page:

http://www.youtube.com/html5

New Toys

2009-11-09T01:55:00.000-08:00

The past week and a half has been a bit more tech filled than usual for me. I upgraded to Windows 7 Ultimate, I got ATI's latest über DirectX 11 compliant video card the 5870 and I've become more active with Twitter on account of the slick TweetDeck client.

The combination of the 5870 and Windows 7 Ultimate has resulted in me essentially having a new system. Despite having a motherboard dating back to mid-2006 this is the second time I've wound up putting off a motherboard/CPU upgrade with a new video card (I'm waiting for ubiquitous USB 3.0 and PCI Express 3.0 before I upgrade my motherboard). My previous video card, the 4870 was an excellent performer but state of the art is a state of change. The 5870 is an amazing product and it definitely falls into the techno-sybarite category. With gaming consoles providing an extremely compelling experience in the form very large communities (XBox Live), large displays (HDTVs) and the ability to ensconce oneself on the sofa, it's not surprising many gamers have eschewed high end PCs for consoles.

The 5870 however proved its power when I pulled out my old Crysis disc. Crysis was a PC game that came out in 2007 receiving lots of press for a couple of reasons - amazing visuals and the fact that probably 99% of people did not have the kind of hardware needed to make Crysis a compelling experience. My guess is Crytek, the developer, had more in mind developing an engine that was capable of realizing the visions of game studios for several years into the future (a.k.a. licensing its engine) and the casualty was everyone that didn't have a PC of the future. I'm happy to say that the 5870 is capable of rendering Crysis @ 1920x1200 with Full Scene Anti-Aliasing (FSAA) and Very High settings. I cheated with my 4870 by playing Crysis on an old 21" CRT I had (since given away) and thus I didn't need to enable FSAA. You see, a graphics card expends quite a bit more cycles applying FSAA. Nowadays I use Dell's excellent 24"2408WFP LCD and my initial reaction to seeing Crysis running on it smoothly was a bit of dissonance as Crysis has had a long standing reputation of bringing hardware to its knees.

Moving along, check out the TweetDeck client, either you get Twitter or hopefully you will realize its utility at some point. I must confess I wasn't blazing trails with Twitter but my coworkers (I wrote about the light bulb going off in a prior post) made me realize how effective it was to ferret out information on a topic that while impromptu was also suddenly very important even if only ephemerally. I will close with an article from the NYTimes I just read that echoes my recent post on Twitter:

http://www.nytimes.com/2009/11/09/business/09link.html?ref=technology

OpenDNS - Your Robust Internet "Phone Directory"

2009-11-05T02:24:00.000-08:00

On this late night (2AM) Comcast's Seattle DNS servers appear to be offline. For the layperson, DNS (Domain Name System) is the magic that happens when you type www.google.com and you're automagically routed to a numeric address (208.69.36.230 - in my case). Every device in your home or office ultimately talks to far off servers through such numbers and not with the text you're used to typing at your browser.

While writing this I received some IMs leading me to believe I've been offline for a couple of hours. Initially I thought it was perhaps hiccups I've seen in the past where power cycling the cable modem clears things up. That wasn't the case this time. I went into my router's dashboard and released my TCP/IP address and asked for a new one by way of DHCP. With each attempt (since I still wasn't getting anywhere) I promptly received the same IP address which gave me a hunch basic network connectivity was available. I then pinged Comcast's DNS server and promptly got a response

Pinging 68.87.69.150 with 32 bytes of data:

Reply from 68.87.69.150: bytes=32 time=55ms TTL=105

Reply from 68.87.69.150: bytes=32 time=54ms TTL=105

Reply from 68.87.69.150: bytes=32 time=57ms TTL=105

Reply from 68.87.69.150: bytes=32 time=54ms TTL=105

Armed with that knowledge I setup OpenDNS. I had actually been running with OpenDNS under Windows XP for quite a while but I moved to Windows 7 last week and hadn't yet bothered. Well, suffice to say, the only reason I can even post this at this hour is because of OpenDNS. In short, OpenDNS rocks, use it.

Why Twitter Is Much More Useful Than You Think

2009-11-01T22:52:00.000-08:00

Even if you could care less about micro-blogging, Twitter is an excellent tool for finding out what is happening in the world as it may immediately pertain to you. It is an observation/technique I picked up from one of the network engineers at work. We were observing what appeared to be a lower transaction rate in England. So a conference call was setup to find out what was going on and yet no one could find anything wrong within the network or backend services servicing England. Then, one of the network engineers searched on Twitter for Broadband England and found many people complaining that they had spotty broadband access by way of one of the equivalents of Comcast over there, i.e. a major ISP. The implications were immediately obvious - we stood a reasonable chance of seeing our transactions affected, i.e. a lower transactions/minute.

So at that point we knew to stop trying to find something wrong within the network as the problem lay externally.

Likewise this evening I was trying to load a game (I recently moved to Windows 7 so I've had to re-download titles) and the Valve Steam client was complaining that the Steam servers were too busy to service my request. This has been the case for a while this evening so I did a Twitter search. And sure enough, I'm not the only one. Click on the image above.

The point being that Twitter can be an excellent tool to feel the pulse of whatever topic you have immediately in mind.

Yeah, There's An App For That

2009-10-24T15:51:00.001-07:00

So how many times have you heard a song on the radio, or a TV commercial (this happened last night) or even a video game (about 20 minutes ago for me) and you think to yourself, “That sounds really good, I think I'd like to own that.”

Well there's an iPhone application called Shazam that has been out for a while but I’ve had occasion to use it this past week across all the situations I just gave as examples:

http://www.shazam.com/music/web/pages/iphone.html

Just hold up the iPhone to the music source (speakers), hit the “Tag” button on the Shazam app, let it sample the music for about 10 seconds and using the power of analysis (some far off servers across the Internet) it tells you what the answer is. In my case (just now it was):

http://www.amazon.com/gp/product/B001W1UY0W/ref=dm_dp_trk8?ie=UTF8&qid=1256422919&sr=301-1

Pjanoo (name of song). You can hit the Preview button on the latter Amazon page and listen in.

This song was featured in the latest trailer the latest Grand Theft Auto 4 expansion. I also heard it some weeks ago during my gym class - the instructor was playing it off her iPod during the cycling class. I heard it yet again in the GTA4 trailer and it crossed my mind “You know, I bet Shazam can help me figure out what that song is.” Indeed it did.

As for the song I heard last night during the TV commercial:

http://www.amazon.com/Walking-On-A-Dream/dp/B0026BI3OQ/ref=dm_ap_trk1

Walking On A Dream by Empire of the Sun.

Bill Gates used to say “Information at your fingertips” eluding to the notion of über powerful desktops (powered by Microsoft of course) housing gobs of information. While some would argue that indeed Microsoft Windows powers many a computer that fetches information off the Net (the host OS for a browser) considering that the HTTP protocol and implications of the greater World Wide Web were not envisioned or spearheaded by MS in any way, shape or form (even after the fact, e.g., cloud computing, no, MS isn't a vanguard there either) and that the iPhone from Apple has seen that dream, imho, take form for people living on the edge of the cloud, i.e., information at your fingertips (in a very, very compelling way), Gates’ vision while materializing isn't being lead by Microsoft.

As for “host OS for a browser” – in light of Apple’s latest quarterly results (best quarter ever; most Macs sold in a quarter), even that could significantly change over the long haul. Case in point, there’s 8 people on my team at work - 5 use Macbooks, 3 of us use Windows (2 XP users, 1 Vista user).

Yeah, there’s an app for that.

Rainy (Computing) Clouds At Microsoft

2009-10-11T10:39:00.001-07:00

Microsoft experienced an event that is not likely to embolden potential customers of its forthcoming Azure cloud computing platform:

http://www.appleinsider.com/articles/09/10/11/microsofts_danger_sidekick_data_loss_casts_dark_on_cloud_computing.html

In summary, if you had Sidekick device through T-Mobile, you have my condolences. Living and working in Seattle I'm aware of some of Microsoft's operations practices. Microsoft's desktop roots, i.e. affinity for shrink wrapped software, has seemingly induced a myopia in how it sees operations. Namely, the belief that if they toss things over the wall, aka contracting, things will get done. The failure of a robust backup process, a.k.a. operations 101, means Microsoft does not take operations seriously (which feeds into things I've heard here in Seattle). This embarrassing turn of events is not likely to instill confidence in Microsoft's forthcoming cloud computing efforts.

All Work & No Play Make Jack A Dull Boy

2009-10-09T22:17:00.000-07:00

Today I went to Fry's Electronics to pickup an Arcade Fight Stick put out by Capcom when it introduced Street Fighter IV earlier this year. The link I've provided to the Arcade Fight Stick shows excellent pictures and is a review. I picked up the smaller one for the XBox360.What some folks may not know is the XBox360 corded controller is a standard USB device and Microsoft readily provides drivers for Windows PCs. More useful still is that game studios have assumed an XBox360 controller when providing joystick support to contemporary games on the Microsoft Windows platform. More specifically games falling under the Games for Windows Live banner. Gone are the days when there was no joystick/joypad standard, de facto or otherwise. A device from one manufacturer might have 4 buttons while another might have 6 and developers often went with the least common denominator making for a less than polished experience. Some games would be released supporting a mouse and keyboard only which is a combination that doesn't always work well. A particular title might afford better visuals on the PC but comes crippled out of the gate having been designed with a game controller in mind due to initially debuting on a gaming console platform. Thankfully those days are over. I had occasion to pick up Street Fighter IV during a weekend "50% off" promotion on Steam some weeks ago. Check out the picture associated with this blog post, I joined a a couple of pics I took with my iPhone 3Gs.

"Duct tape" in six minutes vs. "Beauty" in six months

2009-10-02T08:05:00.000-07:00

I've read Joel Spolsky's blog over the years and his latest post resonates with me in a big way, it's entitled The Duct Tape Programmer:

http://joelonsoftware.com/items/2009/09/23.html

In short Joel says over-engineering dooms software projects and some individuals have a knack for avoiding this pitfall by leveraging disparate technologies, even if it appears to others as a mish mash.

I could not agree more.

I have my share of anecdotes over the years but one of the best ones and most telling was when a coworker with a significant affinity toward Java who had been knee deep in applets for quite some time was asked to write a simple utility to show the differences between two files and send the output to a browser. This is a CGI that would take a duct tape programmer a few minutes to write levering the very common diff utility. However the Java afficionado wanted to build this from scratch, i.e. read two files from Java, write all the logic in Java to compare them. No doubt he would have made a naive tack and his first iteration would likely have shown the first difference in two files not realizing that diff does a very good job of showing other information, not just the first difference. Then he would have spent a considerable amount of time trying to duplicate what the diff utility had already been doing for decades.

Isaac Newton, the father of physics said it best, "If I have been able to see farther, it is because I have stood on the shoulders of giants." Knowing when to leverage the work of others is an important skill in my view. All too often software developers, particularly early in their career have a propensity to see everything as a nail when they wield their hammer (favorite programming language, tool, etc., etc.).

Joel's missive in The Duct Tape Programmer also strikes a cord with me since it eludes to the reason I named this blog MasterCobbler. Specifically the fusion of what initially would seem disparate technologies to some higher end. As a simple example my bringing up Synergy in a previous post to provide one large virtual desktop when having one Windows XP system and a Mac OS X system is at a basic level a very good example of what I'm talking about. The whole is greater than the sum of its parts.

As for some of the examples Joel gave, some contemporary examples are in order. He and I share one thing - we both spent time at Microsoft. I can tell you beyond Microsoft's walls, virtually no one cares about COM, DCOM and that has been the case for quite some time. Once I left Microsoft, the general market had very few people who had zeal for the subject matter and with the passage of (D)COM's hey day, the examples are that much more likely to fall on deaf ears.

Apple Tablet, Windows OS X, Mac OS X

2009-09-30T08:16:00.000-07:00

Apple’s Tablet is not out yet, but here’s more talk:

http://gizmodo.com/5370252/apple-tablet-aiming-to-redefine-newspapers-textbooks-and-magazines

Given my experience with the iPhone 3Gs, very specifically, I’ve enjoyed having the ability to lay down flat on my back and read web pages, if the Apple Tablet is sufficiently light weight such that I can hold it like a book, there’s a very good chance I’ll wind up buying it when it’s out.

On another note, I was at someone’s Vista machine last night trying to diagnose some issues for them. While I like the idea of having an underlying 64 bit Windows OS, more contemporary icons and Windows Media Center (for connectivity to the XBox360), I’ve gotten so accustomed to the appearance of what I've dubbed Windows OS X (see image) at home I don’t like the default appearance of Vista at all. Thankfully Stardock, the company that makes the customization possible, already has a beta of their software running on Windows 7.

I also happen to be in the market for a used Mac Mini, specifically the current high end model:

http://store.apple.com/us/browse/home/shop_mac/family/mac_mini

Except I am not paying $800 for commodity hardware (the parts would probably cost me $350). Lots of people build Hackintoshes but I don’t want to go that route since I’ve read people having issues when a new version of Mac OS X is released, i.e. Apple doing things to dissuade the use of Mac OS X on non-Apple hardware. I don’t need such hassles. I am hoping to find a 2+ GHz Mac Mini on Craigslist at some point, i.e., let someone else bear the cost of lining Mr. Jobs’ pockets with green. So far I have not seen one listed as the current models are relatively new (only a few months old).

If I got a Mac Mini I would setup Synergy and have Windows and Mac OS X be one large virtual desktop across two displays.

The Brewing Storm for Microsoft

2009-09-27T13:32:00.000-07:00

Apparently this nettop (netbook+desktop) from Acer is out now and readily available at Newegg for $200:

http://gizmodo.com/5368829/acer-aspirerevo-nvidia-ion-le-variant-hits-us-shores-for-200

Here’s a more detailed article about the Acer AspireRevo from six months ago on Gizmodo:

http://gizmodo.com/5204432/acer-revo-and-nvidia-ion-hands-on-flawless-blu+ray-playback-changes-cheap-computers-forever

If this kind of hardware becomes crazy pervasive, it could be a big problem for Microsoft in the long term, i.e. the cost of Windows and Office would be significantly more than the cost of hardware. Robert X. Cringely predicted this would eventually become a big problem for Microsoft and it seems with hardware being scaled down so much this is a portent of things to come.

Which is why I have often said that Microsoft made a brilliant move with getting into the gaming console market. Today the XBox360 is a low cost hardware platform whose storage makes it capable of running different programs and in my view Microsoft's entry into the gaming console market was a hedge against this transition. At least I would hope someone inside of Microsoft was sufficiently prescient and saw the potential problem of home PCs being displaced once HDTVs truly became pervasive. Simply because, the top three computing activities in the home are browsing, email and gaming. If you've had occasion to see the web browser that runs on the Playstation 3, it is functional on many web sites. Again, a portent of things to come. Unfortunately for Microsoft every shipped XBox shipped means no license for Windows and Office going alongside it. And Windows and Office accounts for approximately two thirds of Microsoft's revenues.

While the HD playback on the Acer AspireRevo is an issue, things are brewing to remedy this problem:

http://gizmodo.com/5363223/via-pico+itx-motherboard-runs-1080p-video-like-a-champ

While this VIA daughterboard currently is too big to fit into the Acer AspireRevo it happens to be perfect for Netbooks. Now imagine when this VIA PICO hardware winds up being just a single die (chip). Then you are talking about having a $200 PC for your living room capable of delivering content off the Net that does not have the usual hassle factors – cost (the upfront cost and your utility bill going up because of your new space heater), noise (fans) and size (bulk).

It's New & Shiny

2008-09-07T16:12:00.000-07:00

This week I did an initial analysis of the security architecture of Google's new Chrome browser from an operating system standpoint, i.e. how Chrome leveraged Windows to apply the principle of least privilege. This immediately came to mind when I saw page 26 of Google's online presentation of their new Chrome browser.

It also turns out the software architecture in Google's new browser uses an approach that is much more common when writing back end network services. A technique known as privilege separation which page 26 also eluded to.

Privilege separation is merely a technique where the principle of least privilege is applied.

The crux of privilege separation is to create copies of yourself (an .EXE) and strip administrative rights as you do so. It is then those copies then handle inbound requests. So in the case of a web server, the initial instance of the executable (master copy) creates copies and it is those copies that do the actual work of handling inbound HTTP requests. The master copy (the parent of all the children) is simply an orchestrator and never directly interfaces with the outside world where HTTP requests originate. Other network services that follow this model include the venerable OpenSSH server (see image).

The advantage of privilege separation is that if malicious input is fed into a copy handling a request, it becomes much harder to compromise a system since copies have had their administrative privileges stripped. Malicious input's goal is to proxy through whatever is handling a request (copy of an .EXE) to in many cases modify operating system files and directories to nefarious ends. If a copy does in fact serve that end, that would constitute a system breach and what you now have is a serious flaw in the software system. Over the years many buffer overflow exploits in sundry network services have been significantly more severe simply because the technique of privilege separation was not employed. The difference can easily be a system that is compromised and falls into someone else's remote control vs. a much more likely denial of service scenario. Given the choice, I'll take the latter.

Not to say that privilege separation is a panacea. It is not. The technique simply makes a system much harder to breach. And being a technique, nothing says it can't be used on the client side of the equation (a web browser). So while all other web browsers have a single process (.EXE) that manages all browser windows and the tabs within them, Google's Chrome browser creates one copy of itself for each tab where HTML will be rendered and JavaScript executed. As those copies are made, their administrative rights are stripped making it much harder for malicious web content to compromise the computer system where Chrome is executing. Thus Google's Chrome browser applies the technique of privilege separation and immediately affords a level of insulation and resiliency to malware that heretofore hadn't existed in any other browser under Windows XP with its default security settings.

The principle of least privilege in action. Hallelujah.

This is not as big a deal for Windows Vista users but it is a very big deal for Windows XP users. That's because Windows Vista by default does not give users administrative privileges. Under Vista, when you want to do something like install a new application, you're prompted for an administrative password.

Windows XP on the other hand gives full reign (administrative rights) to users out of the box and this is precisely *why* malware has been such a large problem there (much to the delight of anti-virus vendors). Namely, people's browsers acting as proxies for the installation of malware as they promiscuously connect to foreign computer systems on the Internet with administrative privileges. Unfortunately, very few people understand the ramifications of operating in this manner and this extends to most people who make their living with technology.

Google's Chrome browser takes advantage of the Win32 security programmatic interfaces and strips administrative rights as it launches each copy of itself to handle each tab. The magic that allows this under Windows XP is the CreateRestrictedToken system call. The net result is that Chrome under Windows XP makes it much harder for your computer system to be compromised, irrespective of stop gaps measures like anti-virus software.

The level of mitigation is big. How big? Well, the reason Mac OS X has such little malware written for it is because Macintosh users do not run with administrative privileges making the economics of writing malware for Mac OS X a completely different propostion. Malware authors go for the low hanging fruit that are Windows XP users of which there are still plenty. Naysayers often claim, "Mac OS X doesn't have the same market" but that is simply not true. In August of 2007 Apple's market share of laptops hit 17.6% (20% is likely not far off). Apple is now the third biggest seller of laptops behind HP and Dell.

By applying the principle of least privilege Google with its Chrome browser severely mitigates the chances you will contract something like a keyboard logger while innocently visiting your favorite site. Therein lies the danger, increasingly sophisticated and organized criminal groups are breaching the web servers of legitimate businesses to serve up malicious content to unsuspecting users. In many of these cases, malware often comes in the form of keyboard loggers where once installed, absolutely everything you type is transmitted elsewhere in the hope you'll eventually use an online financial account. At which time you unwittingly divulge your username and password despite connecting to a legitimate financial institution (no, this is not phishing). Eventually Electronic Fund Transfers ensue. It's akin to poisoning the water well since as people visit their favorite watering hole (a legitimate web site) they are served up far more than water. As a play on phishing, the term pharming was coined to describe these types of scenarios.

In closing, kudos to Google for leveraging a programmatic mechanism (CreateRestrictedToken) that has been in place for many years. To this day I still do not understand why Microsoft never exposed the power of CreateRestrictedToken with a mass market end user tool or simply embellishing the way desktop shortcuts are created by providing end users with the ability to strip administrative rights in the form of a simple check box. That's because CreateRestrictedToken has been in every copy of Windows since Windows 2000 was released. As I said, many years.

Why 9 > 10

2008-09-07T12:57:00.000-07:00

So I've been resistant to upgrade to Microsoft's Windows Vista for a number of reasons but probably the biggest is consistently observing Windows XP yielding higher Frames Per Second (FPS) with many DirectX titles. Windows Vista introduced DirectX 10 to the world and Windows XP users have had to be content with the older DirectX 9. I had read that Microsoft changed the video driver model for Vista to mitigate the infamous Blue Screen Of Death (BSOD) that has been the butt of jokes for many years but did not investigate further since my plans are to stay on Windows XP for quite some more time.

Here is a shining example of Microsoft being canon fodder on account of BSOD issues, it's a very old Sun Microsystems commercial from yesteryear (before Windows 2000) and pays homage to the late Jacques-Yves Cousteau (the narration). It's quite good!

http://www.youtube.com/watch?v=eNqPTOb31S8

It's also testimony to a long standing problem on Microsoft's operating systems based on the Windows NT kernel introduced in the early 90's. In Microsoft's defense, it can't control how video drivers are written by third parties, but nevertheless, less than perfect video drivers have been one of the leading causes of the infamous BSOD.

So curiosity finally got the best of me as I was talking about DirectX performance with a friend and I decided to investigate why seemingly Vista always loses to XP in sundry benchmarks I've seen in online articles and the printed word. I started with the following Wikipedia article about the Windows Display Drive Model introduced in Windows Vista:

http://en.wikipedia.org/wiki/Windows_Display_Driver_Model

When the Wikipedia article mentioned that part of a Vista display driver lives in userland the light bulb went on.

User space (aka userland) is where applications live, e.g., your web browser, your email client, your IM client, etc., etc. Kernel space is where the code that makes up the OS lives. Both of these terms are commonly used in systems programming.

So the question that beckons, when an application asks the OS (operating system) to do something on its behalf and execution transitions from your application to the OS, how does the operating system protect the stability of the system? After all, you as an application programmer might have just passed some errant arguments that could potentially bring down the system… or on the more sinister side, malicious code could have slipped in through your browser and is making system calls under the hood.

The answer is… with hardware!

The Intel architecture has a notion of “rings” when it comes to executing code (other processor architectures I’m sure have the same semantics, the nomenclature may vary). More on this in a second.

It turns out when executing code, more bits are at play than the usual 32 bits you sometimes hear about, i.e., I’m running 32 bit code versus I’m running 64 bit code. On the x86 architecture memory access is implicitly qualified with additional registers. These registers are called selectors and they happen to be 16 bits. The first 13 bits of a selector are an index into what’s called a Descriptor Table. 2 to the 32nd power is roughly 4 billion so it would seem a 32 bit Intel processor is incapable of seeing more. Such a statement is not accurate of what really is going on. You see, 32 bits refers to the context of a single application, the limits of its addressibility, not that of the processor's.

This means by varying the first 13 bits in a x86 selector register you jump from only having 4 billion memory addresses to:

2^13 (8192) * 2^32 (4 billion)

Which calculates to:

18,446,744,100,000,000,000

But wait, there’s more. The next bit (just one) says whether to go to the Local Descriptor Table or the Global Descriptor Table. So that means you can now double this figure. So you see, the “32 bits” people speak of does not really reflect the mount of memory x86 processors are really capable of managing.

Back to those rings. The last 2 bits in a selector register has to do with rings.

Applications (your browser) run in ring 3 a.k.a. user space. When an application asks the OS to do something, the OS changes last two bits of a selector register and transitions from a higher privileged ring to a lower privileged ring, e.g., from ring 0 (kernel space) to ring 3 (user space). Code running in a lower privileged ring cannot manipulate a selector register to transition back to a higher privileged ring. That’s by design since the OS cannot trust arguments passed to it. From a security standpoint this is sound. You maintain the integrity of the system continuing to provide basic services to other applications and you make malice hard at this particular layer. And there are so many layers… this is why the term computer security is such an overloaded term, has become hackneyed and means different things to different people.

But it turns out these ring transitions have an impact on performance! Any time you have boundary checks, whether they are metaphorical or literal (customs at the US border), things are slower than if you just let things through.

And with Vista’s display drivers partially living in ring 3 (user space), there are likely more ring transitions than the driver model found under Windows XP and its predecessors, hence the 10% to 15% drops (or worse) in performance under Vista.

So while Windows Vista is more stable and mitigates the blue screen of death problems of its predecessors, you get less bang for your buck as far as that screaming new video card goes (at least versus the guy who has Windows XP).

Practical Extraction and Report Language

2008-05-12T06:10:00.000-07:00

Since its debut in the 80's PERL entrenched itself in many computer systems as its power found many uses. The advent of the web applications in the 90's saw PERL spread beyond its roots as the nascent web development community used it to write CGI scripts.

Nevertheless PERL's forte is clearly communicated in its acronym. Perhaps you're after the extraction of information from a large data set based on some arbitrary criteria. Perhaps the only thing you need is to generate a CSV file that will be fed to Microsoft Excel to generate a chart, a form of a report.

The former was the case when a coworker brought me a log file with thousands of lines. He specifically wanted to filter lines in a log file where the criteria was an email address, here's a sample:


./app.comp.log:2008-05-01 08:35:53,288 [WorkExecutorWorkerThread-3] ERROR com.mycompany.events.bo.ejb.NewsSubMDBean  - onMessage: Failed: <root><com.ftd.events.core.NewsEventVO Status="UNSUBSCRIBE" OriginAppName="BOUNCE" EmailAddress="anemail@nowhere.com" CompanyId="123" OperatorId="HARD BOUNCE" OriginTimestamp="1209600000000"/></root>

./app.comp.log:2008-05-01 08:35:53,221 [WorkExecutorWorkerThread-3] ERROR com.mycompany.events.bo.ejb.NewsSubMDBean  - onMessage: Failed: <root><com.ftd.events.core.NewsEventVO Status="UNSUBSCRIBE" OriginAppName="BOUNCE" EmailAddress="anemail@nowhere.com" CompanyId="XYZ" OperatorId="HARD BOUNCE" OriginTimestamp="1209600000000"/></root>

In these two lines the email address is the same, so the second line and any other lines that followed would be disregarded.

If the goal was to simply extract unique email addresses then the ubiquitous cut, sort and uniq text utilities on *NIX platforms could trivially solve the problem. Except the goal was not to extract email addresses, the information around the email address needed preservation. This meant I couldn't apply a cut/sort/uniq solution.

The solution was to employ PERL to save state. I was able to do this in a one-liner:


cat unfiltered.txt |
perl -lane ' m/EmailAddress=\"(.*)\"\sCompanyId/; if ( $have{$1} ) { ++$noise; } else { $have{$1} = 'yes'; print $_} ' > filtered.txt

-lane is one of my favorite command line switch combinations in PERL. a puts every line of input into the anonymous variable, $_. n gives you an implicit while loop to traverse input coming from stdin, l allows you to print information with an implicit newline to help shorten the length of your one liner when sending information back out. e simply is evaluate the expression (code) that follows.

Using a regular expression and a hash allows me to keep track of which email addresses I've already seen. By not printing a line when an email address is already in the hash, I wind up with unique email addresses with surrounding context.

Easy.

Ultramon-itoring the Crysis

2008-04-08T17:56:00.000-07:00

Several years ago I upgraded one of my CRTs to an LCD. The primary benefit of an LCD over a CRT is simply its increased clarity when working with text. Spend a short time on an LCD looking over text and going back to a CRT is painful (on the eyes).

But my CRT has continued seeing use for one specific reason.

If you've ever played a game that has panoramic views, e.g., first person shooters, a CRT has one big benefit over an LCD - fuzziness. "What's that?" you ask. It turns out that the great ability of LCDs to show razor sharp content isn't so great with real time generated panoramic views - the geometry of a scene is filled with noticeable stair step edges of polygons that make up the models that appear in the view.

To combat this problem, over the years Graphics Processing Units (GPUs) from mainstream companies like nVidia and ATI have become much more powerful. With that power, GPUs have increasingly been tasked with performing anti-aliasing to smoothen the edges of an image. The catch in this equation is that your GPU expends far more cycles computationally eliminating the jaggies and this impacts the maximum frame rate you can achieve. It can turn a playable game on a CRT (with no anti-aliasing) into a sloth (unplayble) on an LCD (with anti-aliasing).

A CRT's inability to show razor sharp text lends itself quite nicely to panoramic views that fill the entire screen vs. a small area where your focus is fixated on a word (like the one I just italicized). By turning off full scene anti-aliasing while using a CRT, you spare a GPU a tremendous amount of work. It so happens a CRT’s lack of sharpness for text is great for wide vistas - the natural perspective of first person shooters.

Only thing is, when I've wanted to use my old CRT for gaming, I've had the hassle of making like a monkey and going through GUI motions - right clicking on the Windows desktop, selecting Properties then making the CRT the main display. Then when I'm done, go through this all over again except this time making the LCD the main display.

Yes GUIs are great but not when you need to perform a task repeatedly. Go through this enough times, which I have, and it gets old. Very old. There have been times I don’t even bother firing up a game on account of the hassle factor (but there’s another hassle factor that I’ll get to in a moment, it has to do with the position of application windows on the Windows desktop).

But because of Crysis I’ve been going through the motions as annoying as they are. The game is simply that good. The visuals and realism are stunning. It really ups the ante for first person shooters in a big way.

Before I go further I should point out that I use a very nice little utility program called Ultramon:

http://realtimesoft.com/ultramon/overview/

One of the benefits of Ultramon is a task bar for every extra display you have. Applications sitting in other displays will fill the task bar found on that display versus the task bar of your primary display. This nice touch keeps the task bar on the primary display from getting cluttered. If you work with multiple displays, once you get used to Ultramon in this regard, it's hard living without it.

Another nice Ultramon feature is the ability to set independent wallpapers for each display. Windows XP by default uses the same image on every display when you set your desktop background.

Ultramon also happens to provide an icon in the task bar area that when navigated allows you to quickly set which display is the primary as well as disabling the secondary display. So firing up a game on my CRT, I would use Ultramon’s icon in the task bar to make my CRT the primary display then quickly disable the LCD. Why disable the LCD? Some games like Quake 4 consume a lot of video memory for textures. I found that unless I disabled my secondary display, I could not run it at its maximum display settings (Ultra).

While using Ultramon is faster than right clicking on the Windows desktop and going through the usual motions, even so, the GUI (monkey) way still got old. Very old. Reverting was even even more work - I’ll get to that in a moment.

Then I had some inspiration, I had a recollection that Ultramon provided COM objects with OLE Automation support. This meant I could probably write some VBScript code to automate switching displays and setting things back to the way they were. After reading Ultramon's documentation I came up with these two scripts.

gameCRT.vbs
------

Const POS_ALL = &H7

Set sys = CreateObject("UltraMon.System")
Set Sony = sys.Monitors("2")
Set ViewSonic = sys.Monitors("1")

sys.SavePositions POS_ALL
Sony.Primary = True
sys.ApplyMonitorChanges
sys.SecondaryDisable

enableLCD.vbs
-------

Const POS_ALL = &H7
Set sys = CreateObject("UltraMon.System")

sys.SecondaryEnable
sys.ApplyMonitorChanges
Set ViewSonic = sys.Monitors("1")

ViewSonic.Primary = True

sys.ApplyMonitorChanges
sys.SecondaryDisable
sys.SecondaryEnable
sys.RestorePositions POS_ALL

So the start of gameCRT.vbs makes my Sony CRT the primary display, I apply this new state then I disable the secondary display which implicitly means my ViewSonic LCD. This bears some qualification. I expected to be able to set properties on my two displays then finish with a call to ApplyMonitorChanges but that caused a very annoying problem, at least with a couple of permutations I came up with. Namely, when I re-enabled the LCD later… the positions of the monitors had become inverted so if I moused off the left edge of my LCD (the left display), the mouse pointer would appear on the monitor to the right (the CRT).

Getting around this problem by playing with Ultramon’s object model consumed more time than anything else.

The second script enableLCD.vbs starts off with enabling the secondary display, which is my ViewSonic display on account of the first script implicitly making it the secondary. Then I set the Primary property making the Viewsonic LCD the primary display and again apply my changes with an invocation to ApplyMonitorChanges.

At this point in the script you would expect to be done but it turns out the Windows start menu is still displayed on the CRT even though the CRT is now the secondary display. You would think the Windows Start menu would naturally move itself to the primary display but that wasn't happening. As it turns out, this is not a new problem - I was experiencing the exact same behavior when I would do things interactively.

To solve this problem I disable the secondary display (the CRT now) at which points Windows’ Start Menu gravitates to the LCD (where it was before I decided to fire up a game). Finally, I re-enable the secondary display (the CRT). So reverting things takes additional steps whether you do it interactively or via my VBScript, the steps are logically the same.

That’s why the script has:

sys.SecondaryDisable
sys.SecondaryEnable

It's not immediately obvious why I would disable the secondary display and then enable it immediately afterward but now you know.

Back to back after the ApplyMonitorChanges call.

Ultramon can also keep track of the positions of the windows of applications as you flip flop. Notice this statement in the first script:

sys.SavePositions POS_ALL

And the following statement in the second script:

sys.RestorePositions POS_ALL

WinAmp is very, Very, VERY annoying when you change which display is the primary. The lower half of WinAmp, the playlist, detaches itself from the top half of WinAmp and the two windows get tossed to random positions on the desktop. Meaning WinAmp winds up in an altogether different location than where it was running. Then you have to spend time re-juxtaposing WinAmp's windows. Like I said, extremely annoying.

I also happen to run my CRT at a lower resolution than my LCD. This means when the secondary (LCD) is disabled after making the CRT the primary, applications like Firefox snap to the new smaller dimension of my Windows XP desktop. Unfortunately when I re-enable the LCD and make it the primary, applications do not snap back to their original size and location. Which means you wind up having to resize and re-situate all your applications.

It shouldn't be surprising that there are times you don't even bother firing up a game on account of all this. But with scripting and Ultramon being able to save state, all these hassles are eliminated.

Lastly I created two batch files, gameCRT and lcd which simply map to:

cscript /nologo gameCRT.vbs

cscript /nologo enableLCD.vbs

I have to give kudos to Microsoft for having architected Windows in such a way that it facilitates the development of scriptable objects by third party vendors. Considering the hassles eliminated, I really didn't care what scripting language was at play - VBScript, PERL, Python, etc., etc. In the end I automated the swapping of displays to more easily get my game on with the CRT whenever I saw fit.

Are You A god? Probably. You Just Don't Know It.

2008-04-02T16:11:00.000-07:00

What could I possibly be talking about with a subject like that? Readers will find out soon enough. I've been working on a project. Something for Windows XP/2000 and to a lesser degree Windows Vista.

I submitted the program (with an installer) to Download.com and was given a public availability date of April 11th.

Yes, with this blog entry I'm not writing a "small book". What I created goes beyond catering to a technical readership. It has application to anyone who uses a web browser, including your uncle John who may have gotten his first computer last week because it seems some of his coworkers at the factory got one. Or your aunt Mary because she's into cooking and her best friend Sally whom she bowls with every Friday has gotten tons of recipes off the Internet. Now your aunt Mary wants to dive headlong into the Internet and she's been bugging you to go to Best Buy with her to pick a computer. Or, your grandmother, if she's a vanguard for her age group. But, most importantly, you.

I've been trying to think of a catchy domain. More to come.

Just A Bad Memory

2008-03-24T11:14:00.000-07:00

Java application servers have become pervasive in the last decade as web based applications have replaced old school Visual Basic (VB) applications. Nowadays many corporate web applications are written using Java where the code ultimately runs on a J2EE application server. Java's pedigree is C++ and much to Joel Spolsky's chagrin, Java makes things much easier on developers in one area, memory allocation. In C/C++ a programmer is completely responsible for allocating and deallocating memory. Because of this, bugs at every level, from drivers to operating systems to applications, have found life in C and C++ codebases since the inception of those programming languages in the 70's and 80's, respectively.

Java does away with this onus by completely managing memory. Developers are free to use memory liberally with nary a care. Memory is reclaimed by the Java runtime through garbage collection in the background.

But garbage collection has issues of its own. The biggest is that applications can appear to hang as the Java runtime expends more time trying to reclaim memory than executing code. If you have a time sensitive situation, it can result in outright application failure.

Such was the case when some applications at my employer running under Oracle's application server would die after running an extended period of time. Application failure in the logs was indicated by successive log entries in a short period of time indicating full garbage collection, e.g.:


2624761.994: [Full GC ... 13.5346127  secs]
2624787.633: [Full GC ... 13.4446663  secs]
2624824.282: [Full GC ... 13.4713927  secs]

The indicator here is Full GC. This example indicates full garbage collection had taken place three times in less than a minute. I was asked if successive garbage collection could be monitored, sending an alert if in fact that was the case.

Monitoring a single file is easy, a quick PERL hack would be:

open FH, "tail -f log_file |"
while( <FH> ) {
if ( m/some_string/ ) {
  # Do something when I've seen "some_string"
  # Like send an email
}
}

You could nohup such a script and leave it running indefinitely in the background.

The problem with such an approach is an application server tends to house lots of J2EE applications. You would need a script for each application.

This simple solution does not get around the fact that log files are typically rotated away in the middle of the night. In other words, after a day the log file that is being monitored is no longer reflecting transactions on account of having been renamed with a date extension and moved elsewhere. A new log file is created and the monitor would not be attached to this new file.

Such a solution would also be maintenance prone since a monitoring script would be needed for each application. Therefore with new applications, a script would have to follow in tow and over time the monitoring system may not reflect what is actually running inside a J2EE application server. This shouldn't be surprising since it's quite common for people who write monitors to be separate from application developers.

Furthermore different servers have different applications so now you have a set of monitoring scripts that varies depending on what server you're situated at.

Taking all this into consideration I devised a forking PERL script that gets around all these problems. Forking is not something that surfaces often when writing PERL scripts but its use can greatly simplify some problems. The solution I devised was to have a parent process spawn one child for every log file (associated with an application). The log files are enumerated through a regular expression and the file list is then fed into the PERL script. This way I do not need to keep a list of what applications are running where. In the case of Oracle's application server it prefixes application logs with OC4J~OC4J_.


ls | egrep "OC4J~OC4J_[A-Z]+" | egrep -v ":[0-9]+$" |
nohup perl monitorGC.pl &

And finally the PERL script itself:


#!/usr/bin/perl

$SIG{CHLD} = 'IGNORE';

# Variable a bit of a misnomer at this point, length of time
# for parent to sleep.
$logRotationInterval = 3600; #seconds

# If we see full garbage collection happening more than once in this
# interval, send an alert.
$interval = 60; # seconds

@files = <>;
chomp(@files);

# Process IDs of children;
@pids;

while ( true ) {
    # Loop forever. After one day spawned children will quit since log
    # files are rotated once a day. This main loop will then spawn new
    # children (see comments below).

    for ( $index=0; $index<=$#files; ++$index ) {
        my $pid = fork();
        $pids[$index] = $pid;
        if ( $pid == 0 ) {
            $last = 0;

            open FH, "tail -f $files[$index]|";
            while(  ) {
                if ( m/\[Full GC/ ) {
                    if  ( m/([0-9]+\.[0-9]+) secs]/ ){
                        if ( $1 > 2 ) { # If Full GC is Over 2 seconds
                            m/^([0-9]+\.[0-9]+):/;
                            open LOGENTRY, ">/tmp/monitorGCevent.txt";
                            print LOGENTRY $files[$index]."\n\n";
                            print LOGENTRY $_;
                            close LOGENTRY;
                            if ( $1 - $last < $interval ) {
                                `./sendGCMail.sh $files[$index]`;
                            }
                            $last = $1;
                        }
                    }
                }
            }
        }
    }

    # Sleep while children do their work
    sleep $logRotationInterval;

    # Kill children there's no reliable way to have them exit
    for ( $index=0; $index<=$#files; ++$index ) {
       `kill -9 $pids[$index]`;
    }

    # Kill the tail processes that were spawned
    `killTails.sh`;
}

So there you have it. A PERL script that is application agnostic that monitors successive garbage collection events and sends out an email when such an event is detected.

GotoMySshPC

2008-03-13T19:53:00.000-07:00

One question that surfaces sporadically when managing Internet facing IT systems is What does the rest of the world see? Occasionally after a systems deployment Internet traffic cannot reach some or all of the systems deployed.

This past week for the Nth time in my career this question surfaced. Everyone uses email but few understand the mechanism by which email delivery actually happens. MX records are the key; more on the topic in a moment.

Domain Name Servers (DNS) translate easily read and memorized host names such as www.google.com into TCP/IP addresses, that are not so easy to remember. For example, my Windows XP desktop currently would try to hit one of the following TCP/IP addresses if I were to browse www.google.com:

64.233.167.99
72.14.207.99
64.233.187.99

Quick! Look away! Can you recall any of those IP addresses? If you're like most people, probably not. This particular information was retrieved from my ISP's DNS server. A DNS server therefore is not much different than a phone book.

DNS servers also play the crucial role of facilitating mail delivery. They store different types of information and the type used for facilitating the delivery of mail is the MX record. MX is short Mail eXchanger. When using an email client such as Outlook to send an email, your ISP's mail server which Outlook happens to be chatting with, ultimately has to talk to another computer to deliver the email you just wrote. What computer your ISP's mail server talks to, to deliver that email, is answered by MX records.

One morning mail delivery for a particular domain of my employer was failing and I was called to investigate. On a hunch I did an MX record query for the domain in question and noticed the MX records for the domain had disappeared. This meant that mail delivery to that domain would fail. After further investigation it seems a typo in a DNS configuration file associated with a publish the day prior caused the problem. The resolution was to correct the typo, republish our DNS information and have it propagate across the Internet.

DNS is a hierarchical system. If your local DNS server does not have information that is requested, e.g., what does www.crazydeals.com resolve to?, the request propagates upwards ultimately reaching the top of the hierarchy reaching the root DNS servers of the Internet. The top of the hierarchy is then responsible for disseminating changes is has become privy to, changes that have often occurred at the bottom of the hierarchy. As news of a local change reaches the top, queries from DNS servers at the bottom of the hierarchy percolate to the top reaching the authoritative servers who then pass that information back down about the change that was made in another part of the tree. This information exchange among DNS servers is constantly happening. It usually takes several hours for DNS changes one makes locally to spread across the Internet, sometimes longer.

As the day wore on, I was in the office and wanted to check what my home system saw, i.e., was the DNS change that was made to correct the typo making its way across the Internet.

Thus the age old problem of wanting to conveniently reach my home computer while at the office to check what it was seeing on the public Internet had reared its head again. Once while doing contract work I overheard an individual in a cubicle next to me call his wife to ask her to browse pages to see if they were publicly available. Suffice to say, this problem surfaces on a semi-regular basis over the course of one's career.

Several years ago I had setup a LINUX box that I would reach via SSH. SSH (Secure Shell) is a cryptographic protocol that provides a mechanism to allow issuing commands to a remote computer system. While most people are used to GUI desktops, most back end computer systems are controlled through command line interfaces. If you think the monolithic bank that houses your money along with that of hundreds of thousands other people manages it all by clicking on icons with a mouse, think again. SSH is commonly the substrate by which remote control/management of many large IT systems happens.

One particular command line program that can be used to query DNS servers is nslookup and it comes with any contemporary operating system that connects to the Internet. Every single eye-candy laden Windows and Macintosh system ships with this command line tool. Thus if I could reach my Windows XP desktop, I could use this tool to gauge how far along our change had propagated. At least one vantage point anyway, that of my ISP's DNS servers. Not all DNS servers are public facing, which is why I wanted to see what my home system saw. I wanted to see if the change had trickled down to the private portions of the DNS hierarchy.

There are commercial services that allow one to readily reach one's home computer. One of the better known services is GotoMyPC. My primary issue with GotoMyPC is cost. It runs about $180 for one PC and it is a service, which means count on paying $180 every year.

Another option would be to have the desktop you find yourself at become part of your home network. This is a more general scenario than what GotoMyPC does and is the reverse of what most people do when working remotely. So instead of you connecting to your office via a VPN client, where by though the powers of indirection, you suddenly join your office's network allowing you to work remotely, you would do the same except in the reverse. Your office desktop becomes an extension of your home network. To this end there is an excellent piece of open source software called OpenVPN.

However a VPN solution to the home has two problems. One is that almost all home TCP/IP addresses are part of a DHCP pool so your TCP/IP address changes when the DSL/cable modem sitting in your home has to renegotiate its connection if for any reason connectivity is interrupted, however brief the interruption may be. So if you were to take note of your TCP/IP address, you may find that the next time you were inclined to connect to your home PC, your home network's TCP/IP address is no longer the same. You would be completely out of luck.

A solution to this particular problem exists in the form of an organization known as DynDNS. They allow you to associate your home TCP/IP address with a world wide hostname/domain. This means after registering a domain and using DynDNS to provide DNS services for said domain, you could use something like mypc.mydomain.com when telling OpenVPN to connect to your home network while you are sitting at the office.

Since DynDNS and OpenVPN are free, they are a much more attractive scenario than the pricey yearly cost of GotoMyPC. However you still need to register a domain with a registrar before mypc.mydomain.com (whatever it really happens to be) takes on life. The cost of registering a domain is nominal and about an order of magnitude less per year (vs. GotoMyPC's yearly cost) if you shop around.

There is still one issue that GotoMyPC readily gets around that the OpenVPN/DynDNS solution does not. If your home connectivity is lost and a new IP address is issued, when this information reaches DynDNS' servers it can take several hours before this information (the new IP address) propagates across the Internet. But Murphy's Law may be in full force on a day you really are inclined to reach your home network. If per chance your DSL/cable modem did receive a new TCP/IP address, DNS propagation means it will be several hours before you can connect.

GotoMyPC gets around this by having agent software that chats with their own centralized servers. The agent communicates your home network's current TCP/IP address to centralized servers so DNS is not even involved when trying to reach your home desktop through their service.

If you do go down the DynDNS/OpenVPN route, then like any real business you better remember to renew the domain otherwise you will find your convenient hostname mypc.mydomain.com one day no longer works. It is actually a common problem and various companies including the likes of Microsoft have readily forgetten to do this.

Stepping back for a moment, this all boils down to having information readily in hand, i.e. what's the current TCP/IP address affiliated with my DSL/cable modem's connection to the Internet.

So after this past week I decided to solve this problem once and for all so I could always readily reach my home network, without cost and without hassles. No GotoMyPC, no OpenVPN/DynDNS, no registrars, no accounts with anyone.

Earlier I mentioned that I used to have a LINUX box with SSH running. The primary detractor in continued use was the fact that the TCP/IP address into my home network would change. Eventually I simply abandoned its use. Neither OpenVPN or DynDNS existed at the time.

Once again I would leverage SSH, specifically OpenSSH. While most IT administrators are used to using SSH with *NIX systems or network devices, it turns out OpenSSH can readily be configured to run as a service on a Windows XP/Vista/200x systems, thereby affording remote control/issuance of commands. OpenSSH can be downloaded as part of Cygwin:

www.cygwin.com

Cygwin is a layer of software that makes a Windows system appear *NIX-like. This means, among other things, various open source applications such as Apache's web server (1.3.x) can be compiled and run on a Cygwin equipped system with no changes to the *NIX source code whatsoever. But outside of providing programmatic similarities it also provides ports of back end services such as SSH (OpenSSH). Rather than dive into the gory details of setting up SSH on Windows, go here for an excellent how-to.

Cygwin provides all the command line utilities commonly used in *NIX. People often collectively call a software system LINUX, UNIX, etc. but many of the tools employed in those environments have no intrinsic functionality that ties them to any single operating system. For example, many of the GNU command line utilities have been ported to Windows. The executables are stand alone and run on Windows without any dependencies. When installing Cygwin, all of these tools are available, including command interpreters such as the bash shell.

All this means I should be able to write a script that allows me to retrieve the world wide TCP/IP address associated with my home network. This information however is not stored on any single machine inside of my home network. Like most people I have a home router fronting my network connections. Using my home router's web interface I can readily see what my outside TCP/IP address happens to be so the task is getting this information through a script.

Most people use their web browsers to fetch web pages but the HTTP protocol is very simple and command line utilities exist to do the same. They often form the basis of "heart beating" web applications, i.e., if I can fetch a web page, the web application is still up, if not, send out an email alert. But they have other uses such as this one. I simply want to log into the router, hit the web page that contains my outside TCP/IP address information then finally do something with that information. Easy enough:


#!/usr/bin/bash

while [ true ]; do
sleep 600
wget --http-user=user --http-password=passwd
 http://192.168.1.1/Status.htm
cat Status.htm | cut -c3967-3981 |
 perl -lane 'm/([0-9.]+)/; print $1' > currentIP
diff currentIP lastIP
if [ $? -ne "0" ]; then
# Send mail about IP change
cscript sendIPAddressViaMail.vbs
cp currentIP lastIP
fi
rm Status.htm
done

My home router performs basic access authentication and I can pass the requisite username/password through command line arguments to the wget utility program when I retrieve the web page that contains the TCP/IP address my DSL modem negotiated. In the case of my LinkSys router that happens to be:

http://192.168.1.1/Status.htm

Initially I used wget in an ad hoc manner and it by default it stores the page fetched in a file. I noticed the page that had been returned from the router had no line breaks so I fired up EMACS and navigated to the column where the TCP/IP address affiliated with my outside connection starts. Noting the column offset I scripted cut to pluck 15 characters (max possible TCP/IP address string length). Finally I filter the string through PERL so that only characters making up a TCP/IP address would be in the final string that I output to the console.

I store the extracted TCP/IP address in a file named currentIP and compare it with the diff command to the last IP address (lastIP is initially setup by hand). If there are no differences in the files (determined by checking diff's exit code), this indicates my TCP/IP address has not changed so the script goes to sleep for another ten minutes (600 seconds).

If the files are different then things get a bit more interesting. How do I communicate this information? Most *NIX administrators use the mailx utility to send email via their scripts. The problem with mailx is that it assumes there is a local mail server running and running your own mail server is a can of worms onto itself. And honestly I'm not inclined to run a mail server on my Windows desktop. Rather than dive into such issues, I leverage Microsoft Outlook. Knowing that the Microsoft Office applications can be automated with VBScript, I concocted the following script that is executed through the Windows command line tool cscript:


ESubject = "IP Address change"
SendTo = "mymail@someWebMailAccount.com"
Ebody = "IP Address change"
NewFileName = "D:\cygwin\home\mariop\currentIP"

Set App = CreateObject("Outlook.Application")
Set Itm = App.CreateItem(0)
With Itm
.Subject = ESubject
.To = SendTo
.Body = Ebody
.Attachments.Add (NewFileName)
.send
End With
Set App = Nothing

The script sends a text file that contains the TCP/IP address my DSL modem last negotiated as an attachment to my web based email account. This way I can always just browse my web mail to find out my home network's current TCP/IP address. If my TCP/IP address changes, an email will be sent out within ten minutes.

There was one last hurdle to this solution. In 2002 given the prevalence of VBScript based worms in years prior, Microsoft changed Outlook such that automating the sending of emails was not possible without confirming email sends. Outlook will now issue a pop up asking whether or not to allow email being sent through a script. This a major fly in the ointment since I need this to be unattended since after all I'm not at my home PC. After some googling I came up some freeware:

http://www.contextmagic.com/express-clickyes/

The latter page shows the pop up that surfaces if email is sent via VBScript.

There you have it, with all this in place I'm assured of always being able to reach my home desktop. No GotoMyPC, no OpenVPN/DynDNS, no registrars, no accounts with anyone.

It turns out the solution I employ can be used in conjunction with OpenVPN eliminating the need for DynDNS and having to register your own domain. However I prefer SSHing since this allows the computer I'm working on to maintain its local context. I can readily switch between what I'm doing at work and what I might want to do at home. Namely, the machine I'm working at doesn't become part of my home network and suddenly local/work resources are unavailable to me.

If you're at all familiar with SSH's abilities you can tunnel various application protocols such as Microsoft's Remote Desktop. This means I can reach my graphical Windows XP desktop as the need arises. Which is exactly what GotoMyPC provides. Except in my case, without the $180 yearly cost.

When I establish an SSH connection to my home system, I do something like this using the SSH binary that is part of Cygwin on my work machine:

ssh -L 3390:localhost:3389 my_home_ip_address

After I've logged onto my home system instead of giving a machine name to Microsoft's Remote Desktop client while I'm sitting at the office, I simply specify:

localhost:3390

And before too long I get a login to my home Windows XP desktop while sitting at the office. The entire conversation between work and home computers is encrypted.

GotoMySshPC.

Mod-direction

2007-12-18T06:28:00.000-08:00

If levels of indirection in our lives were as prevalent as those found in information technology, life would be pretty tedious and it would be an awful lot of work to get even small things done. Yet levels of indirection are par for the course in technology. The abstractions they engender form a patchwork that give us the applications we are accustomed to using. Everything from desktop applications to the World Wide Web.

For example we speak of 32 bit processors and increasingly 64 bit processors and it is mostly understood by those in technology that this delimits addressable memory. Which means that in the case of your 32 bit system (probably; at the time this was written) it can address 2^32 bytes of memory or more specifically 4 gigabytes in the context of a single application. In the case of a 64 processor this rises to 1.84467441 × 10^19 bytes. That's:

18,446,744,100,000,000,000

So a 4.6 billion times increase over what a 32 bit processor can address.

But it turns out that other "bits" dictate the bytes that make up the data and code associated with our applications. The Intel x86 architecture for example has registers called selectors which comprise 16 bits. The first 13 bits are actually an index into a table the processor keeps, the next bit dictates which table to do a lookup against. A single bit has two states, 0 or 1, which means that there's two tables. These happen to be called Global Descriptor Table and Local Descriptor Table. 2^13 means 8192 possibilities which means there's at least 16,384 initial paths before the 32 or 64 bits that people typically speak of start to take context - in the form of either code to be executed or data to manipulate. To make things very lucid for a 64 bit processor this means:

18,446,744,100,000,000,000 x 16,384 logical paths to code and data

To quote Sir Isaac Newton, "If I have been able to see further it is because I have stood on the shoulders of giants."

Much of we take for granted in our every day lives is attributable to "behind the scenes" machinations seemingly akin to caricatures of of mouse trap contraptions that have played themselves out in pop culture from cartoons to board games. From the power grid system to water flowing through our lavatories to the Internet there is much ado with the conveniences we are accustomed to with the flick of a finger.

A router in a home masks the fact that a user has several machines behind a single TCP/IP address. This many to one relationship is a level of indirection. This indirection, redirection or in the case of malice, misdirection (think botnets), happens at a level that both endpoints are oblivious to, e.g., a web browser and a web server.

Things get interesting when decision making (indirection) happens at the the initial stage of browsing a web site, which is mutually exclusive of the indirection happening at your home router. It may turn out that the initial chatter your web browser conducts when going against your favorite web site is really against a load balancing device that makes decisions based on the URLs you are requesting. So when you hit www.my-favorite-site.com/search/query.jsp, the device keys off of search in the URL and always sends these page requests to one or more web servers dedicated to searching the knowledge space of that site. To this end, products from f5 are well known in the industry. This is often known as layer 7 load balancing. Layer 7 is the top most layer of the OSI model of networking and is called the application layer since that layer is closest to the abstractions that make up an application, in the example given, the request to a search page for information.

A home router works at layer 3. All these layers are levels of indirection that build on top of each other to facilitate the applications that are familiar to any contemporary Net user, from IM to web browsers to Bittorrent clients.

Another potential decision point for web servers is the kind of HTTP method a client is making. The HTTP protocol allows a client, such as a web browser to make different types of requests to a web server. An overwhelming majority of requests made against sites are usually GET requests by web browsers and it means what it insinuates get me a page. The second most common HTTP method is the POST method. This method is usually used with forms, so anytime you've entered your name, address and credit card and hit the Buy button, odds are overwhelming that an HTTP POST method was at work.

When the HTTP protocol was designed in 1989 the world was a simpler place, the issues of phishing, email spam and other rogue forms of misdirection were yet to surface. In 1999 with revision 1.1 of the HTTP protocol the HTTP TRACE method was added. Having done software development in the past, the motivation is clear, it was designed as a debugging tool. The method simply has the web server echo a request that was sent to it. Sounds benign enough except that if your web browser has cookies associated with the path sent to the HTTP TRACE method, the web server will readily echo the cookie values. Again this sounds benign enough except in the contemporary world of rogue JavaScript (embedded in a malicious HTML email that happens to be spam), the HTTP TRACE method in conjunction with the XMLHttpRequest object can be used to read cookie values and then send them off to someone who just might then go into your bank account and relieve you of your finances. This forms the basis of a Cross Site Scripting attack.

The XMLHttpRequest object in JavaScript forms the basis of Web 2.0 applications, i.e., applications that are very dynamic in nature and feel like a local desktop application. You could turn off JavaScript but you will find out soon enough that it is a lost cause. Most sites rely on JavaScript to provide the experience that end users are accustomed to.

One problem in the management of IT infrastructure and the applications that live on that infrastructure is time. In time, every computer system that is deployed will degrade and become a legacy system.

Any computer system has two factors working against it.

The hardware technologies employed will become antiquated and may not be up to the ask with the growth of an organization. If the system is up to the task other factors over time readily contribute such as replacement parts or in some cases, the vendor of the original hardware ceasing support or going out of business. Some systems are labeled legacy systems but very much still form the crux of what is happening in the here and now. The Internal Revenue Service grapples with the problem of having a well entrenched legacy system.

With the software residing on a system, a wide range of factors contribute it moving to legacy status. Everything from employee attrition (waning knowledge base) to business requirements not being met with the software initially deployed to finding people with skill sets that can make changes to the system.

The factors and their weight will vary widely but just as time weighs down on our knees, time will weigh down any computer system from both the hardware side and the software side making continued use a questionable business proposition. Either in terms of the opportunity cost of lost business or the prohibitive cost of the status quo. In the case of the IRS, the government being what it is, it has resources (your tax dollar) that the private sector could not muster, a.k.a. bankruptcy.

This past week I ran into one of these legacy systems. An Internet facing application server employing JBoss. JBoss is intended to run Java code that makes up web applications. RedHat acquired JBoss in April of 2006 for $350 million. RedHat is well ensconced in the IT industry and JBoss is still well supported so what made this system a legacy system? The version of JBoss running on the server in question was dated to March of 2003 and staff that put it into use where no longer with the organization. More to the point there was little documentation. This is classic case of attrition making a system difficult to support and why managers should be big fans of knowledge transparency in the forms of wikis and allocating time up front to document systems. My role in this case was strictly that of system administrator of the underlying LINUX OS.

This Internet facing application was flagged by auditors for a Cross Site Scripting vulnerability on account of the web server servicing the HTTP TRACE method. I was tasked with seeing if the system could be reconfigured to turn off servicing HTTP TRACE requests.

As I performed a discovery process I learned that the first tier that responds to HTTP requests is a complementary piece of open source software to JBoss called Jetty, an open source web server written entirely in Java. After I ferreted out where Jetty's configuration file lived, through online investigations I discovered that the version of Jetty included with the version of JBoss found on the production system did not have the facility of turning off the HTTP TRACE method. In fact, the version of Jetty employed was one minor revision number off from having the ability to turn off HTTP TRACE.

A quick initial assessment of the situation purely as a function of the application served up as well as the application server itself (JBoss/Jetty) seems to leave only difficult choices. If the application server software is updated, it could very well break the underlying application. As alluded to earlier, part of the problem with legacy systems is knowledge transfer, or lack thereof. Outside of a cursory inspection of this Internet facing web application such as logging on to check base functionality, little was documented, such as a regression test plan or even the tools to carry out such a regression. Perhaps an upgrade of the underlying application server would go well initially (software starts) but application functionality will break (what end user's perceive). Rewriting the application is not realistic either. The application's entry point was flagged during an audit so more than likely remediation is expected sooner rather than later. In addition, asking managers to expend resources to rewrite applications impromptu is not likely to garner support (think budgets), especially if the scope of the application is large. Nor is it likely to thrill tech people who may be on time lines with other projects (see point 8).

Indeed this seems like a very thorny problem with choices that entail lots of risk. Unless that is, you happen to be familiar with the various levels of indirection and their layers and how they play among themselves but even more importantly the tools to manipulate them.

The Apache web server is an amazing piece of software. Despite years of Microsoft giving Internet Information Server (IIS) away for free and Steve Ballmer's hot air of calling one operating system that popularly runs Apache a cancer, i.e. LINUX, it still remains the most popular web server in use on the Internet today. Rather than take my work for it (or not), visit Netcraft which offers a tool that allows you to see what web server your favorite web site is using. The link provided shows the web server platforms behind sites that people were curious about the most - observe that Microsoft's platform is barely on the radar (12/2007).

This is no coincidence. One of Larry Wall's mantras for his popular PERL programming language when he designed it was make the easy things easy while making the difficult things possible. It is a philosophy that shows up consistently with the proponents of open source technologies. It is one thing to be the end user of a site, say Amazon, that may be a complex heterogenous mix of computing platforms, ignorance as they say is bliss, but it is quite another if you must administer the platforms that comprise such a site. The more degrees of freedom that exist to make the system pliable, the greater the ability to adapt to situations as they arise, planned or unplanned... such as when an auditor is complaining.

What makes Apache so powerful specifically are all the modules of extensibility that come with the system 'out of the box'. They are often called mods for short. Being part of the open source community the Apache web server has engendered an active developer community that affords administrators great flexibility with configurations but also in the manipulation of HTTP traffic.

Manipulation of HTTP traffic is one powerful use case for Apache. In this capacity, an Apache web server instance never actually hosts any web pages but is used as a traffic cop to redirect HTTP traffic based on rule sets giving the illusion to the end user of a cohesive web site but in actuality the site may be an amalgam of different web servers.

The Apache modules mod_proxy and mod_rewrite provide these facilities and extensive use cases and the minutiae of the semantics of the rule sets are beyond this write up. Suffice to say they can be used to solve the thorny problem of turning off the HTTP TRACE method.

The solution is simply to have the original web server that fronts the offending application to listen on a non-standard TCP port such as 8080, which is not visible to the outside world on account of a firewall, have Apache listen on the well known port (80) then with a mod_proxy/mod_rewrite rule set direct traffic based on the HTTP method. If a request in the form of HTTP TRACE method comes in, simply deny it.

This is easily done through:


RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]

RewriteRule ^/(.*)  http://localhost:8080/$1 [R,P]

The RewriteCond statement looks for the TRACE method and the RewriteRule after it causes Apache to return a 403 Forbidden page. The second RewriteRule is a catch all that simply has Apache delegate all other HTTP traffic to the server listening on port 8080 on the same system Apache resides, which in this case is the offending JBoss/Jetty server that will gladly service HTTP TRACE requests. However Apache configured as such will filter these TRACE requests so all that the auditor sees is a result that nullifies the previous observation that this particular Internet facing web application services the HTTP TRACE method. And thus with a deploy of the Apache web server and the addition of four statements in its configuration file, no blown budgets, no interrupting developers.

In the interest of a balanced viewpoint it turns out that if you have a contemporary application load balancing device such as those from f5 the filtering of the HTTP TRACE method can happen there. But it turns out in this case the application was not fronted by such a device. If such a device does exist, the politics at play within an organization may make it more work to involve other teams to make network changes rather than to reconfigure software on a single server that ultimately receives the offending requests.

A better decision point however is the number of points of potential change. It is more cost effective to manage access control from a central location such as an f5 device than visiting multitudes of boxes to make configuration changes if the Internet application is backed by an entire web server farm. Since that was not the case here, an Apache solution was employed.

To quote Arthur C. Clarke, "Any sufficiently advanced technology is indistinguishable from magic." If these tiers of indirection are not on your radar, yep, magic.

As easy as -p -i -e

2007-08-09T07:48:00.001-07:00

Before the advent of the World Wide Web and the decoding of the human genome, PERL had already ensconced itself in the IT world amongst *NIX system administrators almost ten years prior. For good reason. PERL provides a more complete feature set than the Bourne Shell and/or sed/awk. Then and now, all of these tools are commonly used by system administrators to automate repetitive tasks.

One of my favorite invocations of PERL is with the command line options -p -i -e. This particular combination allows for the in place editing of files, namely searching for one string and replacing it with another. Like so:

perl -p -i -e 's/original_text/replacement_text/' configuration_file

You can even qualify the -i parameter to back up the file being modified:

perl -p -i.bak -e 's/original_text/replacement_text/' configuration_file

The original file will continue to live on as configuration_file.bak.

Recently, a coworker was faced with changing a configuration file to point from one database to another on six production instances of Oracle's Application Server. Each instance in turn housed ten applications. This meant he would need to go edit sixty configuration files or use Oracle's web based administration tool and drill into it sixty times. Irrespective of which of these methods you employ, the process is tedious and error prone.

Armed with such knowledge, cobbling a solution for my coworker's plight did not take long. Logging into any number of machines via a script is one reason the Secure Shell(ssh) exists. Since I administer the boxes the changes needed to be done on, I had the requisite private cryptographic key to log into all of them without being prompted for a password.

This problem beckoned for automation since I could readily traverse all of the machines from a central point and invoke commands. In summary, I wanted to log into each machine, find the configuration files of interest, do an in place substitution to point to a different database.

Here is the solution for saving my coworker from firing up vi sixty times across six machines:

for i in `echo "mach1 mach2 mach3 mach4 mach5 mach6"`; do
ssh $i 'cd /oracle/10gR3; find . -name "data-sources.xml" | xargs perl -p -i.bak -e "s/oracle/microsoft/" ';
done

The for loop is a shell construct executed on the system that serves as my jumping point to reach each system where I employed ssh-agent to avoid password prompting. Ultimately the heart of what I am doing on each machine is this:

cd /oracle/10gR3
find . -name "data-sources.xml" | xargs perl -p -i.bak -e "s/oracle/microsoft/"

I situate myself in the base directory of the Oracle application server, find all the copies of the configuration file in question, data-sources.xml, then use xargs to invoke PERL to do in place substitution against each and every instance of said configuration file. Using find with the -exec option and removing the intermediate xargs invocation works equally well.

So now, instead of wasting an hour (or more) editing sixty files, we both can hit the golf course an hour early.

This experience underscores the potency of using several seemingly disparate tools to solve a problem - very much the *NIX way of doing things. Now if only I could do these kinds of things "out of the box" under Windows...

An impromptu GUI for strace when monitoring a forking server

2007-08-07T11:57:00.000-07:00

I used to have a coworker named Tom where it was joked among us that he had only two possible answers to any technical hurdle posed to him - strace or tcpdump. As it turns out, more often than not, he was right.

strace gives visibility into OS system calls made by a running process. tcpdump allows sniffing of network traffic and is often used to ferret out networking issues. For monitoring an application as it interfaces with a host OS and/or its interactions over a TCP/IP network, these two tools are indispensable.

I once deployed a web application that during startup failed on account of a missing file. As best as I could tell, the file in question was present. Count on strace to get to the bottom of things. It turned out to be a simple misconfiguration - the file lay situated in a parent directory of the code instead of a local directory. This was easily divulged when I ran strace and could readily see all file open operations during the start up phase. What I was able to readily infer from its output was much more meaningful to me than Java code complaining that it could not open a file and then immediately terminating (but not divulging the path of the file it was trying to open).

Strace is powerful but its output can be quite noisy if the application being monitored has a high level of activity. If per chance monitoring a forking server such as Apache or Postfix is desired then strace really gets noisy. Making sense of the "spaghetti" that is returned back from system calls associated with a parent/child process tree is not fun. While it is possible to have strace latch onto a specific child process, errant behavior that is occurring may not be happening in the process being monitored but instead a non-monitored sibling process. Therefore using strace to have complete visibility into a forking server where the output is readily digestable is problematic.

I found myself precisely in this boat as I wanted to monitor the file activity of a Postfix instance running under Solaris. In Solaris, strace's equivalent is truss. After not too long here is the solution I devised:


lsof |
grep TCP |
grep smtpd |
awk '{print $2}' |
sort |
uniq |
perl -lane ' system("/usr/X/bin/xterm \"-sb\" \"-sl\" \"1000\" \"-e\" \"/usr/bin/truss\" \"-p\" \"$_\" &") '

lsof is a tool that will list all open file handles on the operating system, including network file handles. I specifically was looking for processes that had TCP/IP network connections (first filter). Then I looked for smtpd which is the name by which Postfix is listed in the process table (second filter). After which I used awk to pluck all the Postfix process ids which are in the second column of lsof's output (third filter). I sorted these (fourth filter) and removed any duplicates (fifth filter). Finally, this list of process ids was fed into a PERL one liner (sixth filter) that spawns xterms with the -e option. This option launches an application inside of an xterm, in this case truss.

Employing Cygwin's X Server port on my Windows XP desktop, an xterm appeared for each Postfix process running under Solaris (see image above), with truss showing me all the system calls for each of them in real time. Outside of the fact that this approach yields information that is readily consumable for spur of the moment diagnostics, another advantage is that when a process terminates, the corresponding xterm housing truss monitoring that process disappears off the desktop. Which is good visual feedback for taking action depending on the context of the situation.

For PERL to correctly interpret my intentions, I had to escape the double quotes. Furthermore when xterm is launched in this fashion, double quotes are necessary to delimit the command line arguments. So the sample code is harder to read than if one were to interactively fire off an xterm interactively against each and every process like so:

xterm -sb -sl 1000 -e truss -p process_id_in_question

This statement says to fire up truss inside of an xterm against the process id specified after the -p argument with a scroll bar (-sb) and with a 1000 line scroll back buffer (-sl 1000).

Canonically, child processes of forking servers are ephemeral. That is, a child process on a forking server will handle a given number of requests and then terminate. This design philosophy insures that one child cannot live long enough to consume all available system resources, either by accident or malice (serving as a proxy for Denial of Service attacks).

Therefore coming up with a script that hard codes process ids is pointless as their number and ids will always vary over time. Whereas this technique fetches process ids on the fly and couples this with the ability of an xterm to house an application (truss in this case), the end result being an ad hoc GUI for stracing/trussing a forking server.