The Host Network Stack

This post is a collection of useful articles/videos that I’ve collected about networking on XenServer and Linux.



As you can see, there are a multitude of elements to consider when looking into host networking issues for a Linux VM running on XenServer (which is Linux underneath the covers anyway).

the grep is a lie

grep is a wonderful tool for digging through logs on specific issues, but there are a few cases when people misuse it and claim the logs don’t have the answers when grep didn’t yield an answer.

Here’s an example of Rails application logging from Ruby on Rails Guides:

Processing PostsController#create (for at 2008-09-08 11:52:54) [POST]
  Parameters: {"commit"=>"Create", "post"=>{"title"=>"Debugging Rails",
 "body"=>"I'm learning how to print in logs!!!", "published"=>"0"},
 "authenticity_token"=>"2059c1286e93402e389127b1153204e0d1e275dd", "action"=>"create", "controller"=>"posts"}
New post: {"updated_at"=>nil, "title"=>"Debugging Rails", "body"=>"I'm learning how to print in logs!!!",
 "published"=>false, "created_at"=>nil}
Post should be valid: true
  Post Create (0.000443)   INSERT INTO "posts" ("updated_at", "title", "body", "published",
 "created_at") VALUES('2008-09-08 14:52:54', 'Debugging Rails',
 'I''m learning how to print in logs!!!', 'f', '2008-09-08 14:52:54')
The post was saved and now the user is going to be redirected...
Redirected to #
Completed in 0.01224 (81 reqs/sec) | DB: 0.00044 (3%) | 302 Found [http://localhost/posts]

Grepping for “learning” will give us just a peek but there’s much more information to be found in the full request.

# grep learning applog.log

 "body"=>"I'm learning how to print in logs!!!", "published"=>"0"},
New post: {"updated_at"=>nil, "title"=>"Debugging Rails", "body"=>"I'm learning how to print in logs!!!",
 'I''m learning how to print in logs!!!', 'f', '2008-09-08 14:52:54')

If you know the exact format of your applications log messages, you can use output context flags within grep (-A -B and -C). However, a lot of the time the exact number of context lines needed is unknown or a particular stack trace could have a varying length.

Rails applications aren’t the only ones – the logging module within Nova also falls to this same issue. Common Log Format seems to get around the problem, but many modern applications or ones in debug mode have multiline/transaction-ID logging which make sole reliance on grep a bad decision.

My preferred technique: Use grep to determine which log file to open in less. Then, use the pattern search within less that I grepped and take a look at the clues provided in the context. Sometimes it’s as simple as two lines later you’ll see a SIGTERM, but you wouldn’t have grepped for SIGTERM.

Another tip with less and pattern matching: if you have a large file and you know relative to the file, your search string is toward the bottom hit G to move less to the bottom of the file, then do your /pattern search, but then press N to find the previous result.

One last thing: if you haven’t given zless/zgrep a try on compressed files, they’re worth their weight in gold.

Gem in the VMware vCenter Converter 4.2 for vCenter Server Release Notes

It’s been a while since I’ve had time to post, but I couldn’t keep this to myself:

From the VMware vCenter Converter 4.2 for vCenter Server 4.1 Release Notes:

What’s New
The VMware vCenter Converter 4.2 is a substantial upgrade from vCenter Converter 4.1 and includes the following new functionality (previously found only in vCenter Converter Standalone 4.0.x):
Physical to virtual machine conversion support for Linux sources including:
Red Hat Enterprise Linux 2.1, 3.0, 4.0, and 5.0
SUSE Linux Enterprise Server 8.0, 9.0, 10.0, and 11.0
Ubuntu 5.x, 6.x, 7.x, and 8.x

This sounds too good to be true. I upgraded my lab environment and confirmed:  you can now do scheduled Linux P2V.

The lab setup was ESX 4.1, vCenter 4.1, and vCenter Converter 4.2 for vCenter Server with a RHEL4.7 AS server imported.