malayter.com

Bufferbloat on a 3G network

2012-02-06T10:58:00.000-06:00

I've been experiencing terrible network performance lately on Sprint's 3G (EVDO) network in downtown Chicago. I have both a Sprint Mobile Broadband card for my laptop and an iPhone 4S on Sprint's network. Sprint performance used to be fantastic compared with AT&T and Verizon mobile data networks in Chicago, but the introduction of the iPhone on Sprint seems to have caused some capacity problems. The worst spot I've come across seems to be the Ogilvie train station.

I decided to run some diagnostics from my laptop in the area during a busy rush hour. I expected to see that the network was just hopelessly oversubscribed, with high packet loss. This is a very busy commuter train station, and there are probably tens of thousands of 3G smart-phones in the vicinity at rush hour. There's also lots of mirrored glass and steel in the high-rise buildings above - basically it's the worst "urban canyon" radio environment imaginable.

However, some simple ping tests from my laptop broadband card showed almost no packet loss. What these simple tests revealed instead was a problem I previously thought carriers knew to avoid and only really affected consumer devices: "buffer bloat".

Click to embiggen

The data in this chart was collected on Friday, 3 February 2012 starting at 5:03 PM. I ran a simple ping test from my laptop to the first-hop router on the 3G connection, so I was essentially only testing the "wireless" part of the network. I presume that Sprint has fiber-based backhaul from their towers in downtown Chicago that have plenty of bandwidth.

The minimum round-trip time observed was 92 ms, which is similar to what I see on 3G networks in uncongested locations. However, ping times varied wildly, and the worst round trip time was over 1.7 seconds. . An RTT this long is disastrous from a user experience standpoint. It means, for example, that connecting to a HTTPS-enabled site takes nearly 7 seconds before the first bit of the HTML web page is transferred to the client.

There was no other network activity on my laptop at all, so this insane result seems to have come from the 3G network itself. It looks to me as though Sprint has massively over-sized the buffers on their wireless towers. There is really no excuse for this at all given the recent attention buffer bloat has received in the networking community. I can't think of any circumstance beyond perhaps satellite communications where holding on to an IP packet for 1.5 seconds is at all reasonable.

Now, I suppose the problem could be caused by upstream buffering on the laptop. But as I said there was no other activity, confirmed by the wireless card's byte counters. Even if a flow-control mechanism in EVDO was telling my laptop not to transmit, or even telling it to re-transmit previous data, there should not be 1.5 seconds of buffering in the card or its drivers. An IP packet 1.5 seconds old should just be dropped.

I plan on doing more testing in the near future, but I have to ask the mobile networking experts out there: am I totally mis-interpreting the data from this admittedly simple test? Is this buffering something inherent in CDMA technology? Can anybody think of a test to see if it is the OS or driver buffers holding on to the packets for so long (I don' think Wireshark would work for this). Obviously I don't have access to hardware-based radio testing equipment, so software tests are all I can really do.

Assuming the problem is actually the downstream buffers in the tower, Sprint really needs to adjust their buffer sizes and start dropping some packets to make their 3G network usable again.

Technical details of equipment used in the test: Dell D430 laptop, Windows 7 current with all patches and service packs, Dell-Novatel Wireless 5720 Sprint Mobile Broadband (EVDO-Rev A) card with firmware version 145 and driver version 3.0.3.0.

What the Internet looks like in 2012

2012-01-06T13:42:00.000-06:00

Click to zoom

This is a graph of links between visible Autonomous Systems on the Internet generated from public BGP routing tables early on 1 Jan 2012. Each link of each AS path in the BGP data is represented as an edge, with duplicates removed. The data was then graphed using the twopi layout tool from Graphviz. Links to the top twelve most-connected service provider networks are highlighted in color, with all other AS links in white.

I'm struck by the sheer density of connectivity on the modern Internet. Each of the 94865 lines on this graph represents at least one physical link between organizations. But in the case of larger networks that same thin line might represent dozens of routers and 10 Gb/s fibers at many locations throughout the world.

It certainly looks as robust as originally intended, but also chaotic and disordered. Surely no government, organization, or evil genius bent on world domination could possibly control all those links. The sooner our politicians figure that out, the better.

H.264 versus WebM

2011-01-12T22:00:00.018-06:00

There's been a lot of noise recently about Google's supposedly "free" WebM video format versus the widely-used (but patent encumbered) H.264 video format used by Flash, Apple devices, Blu-Ray players, and just about everything else.
The best H.264 encoders perform better than WebM's VP8 codec based on the objective SSIM metric, and the consensus is that H.264 video in general looks "better" at the same bitrate than VP8. But how much better, at "web video" resolutions and bitrates?
I took a number of widley-used HD test clips (which are designed to "stress" video codecs) and concatenated them into one test video. I then encoded the result at 640x360 resolution at 500 kbps.
First, the H.264 sample. I used x264 as it seems to be the best H.264 encoder around.
Your browser doesn't seem to support the HTML5 video tag for MP4 videos. Here is a link to the H.264 sample video.

x264 core:110 r1820 fdcf2ae

x264 --preset veryslow --tune film -B 500 --threads 0 --pass 1 -o all_vids_360p_500k_h264hi31.mp4 all_vids_360p.y4m
x264 --preset veryslow --tune film -B 500 --threads 0 --pass 2 -o all_vids_360p_500k_h264hi31.mp4 all_vids_360p.y4m

pass 1: 62.66 fps, pass 2: 32.48 fps, total encode rate: 21.39 fps, final bitrate: 502.10 kbps

Next, the WebM sample, encoded with the vpxenc tool provided by the WebM project. Your browser doesn't seem to support the HTML5 video tag. Here is a link to the WebM sample video.

vpxenc WebM Project VP8 Encoder v0.9.5

vpxenc all_vids_360p.y4m -o all_vids_360p_500k.webm -p 2 -t 8 --best --target-bitrate=500 --end-usage=0 --auto-alt-ref=1 -v --minsection-pct=5 --maxsection-pct=800 --lag-in-frames=16 --kf-min-dist=0 --kf-max-dist=250 --static-thresh=0 --drop-frame=0 --min-q=0 --max-q=60

pass 1: 71.36 fps, pass 2: 8.88 fps, total encode rate: 7.90 fps, final bitrate: 508.298 kbps

Now, the WebM sample looks pretty good, except for the "tough spots" such as the flames, water, and fade transitions. It also seems "softer" overall than the H.264 video. Clearly, VP8 has improved with the latest 0.9.5 release, but it is still not close to H.264 at "web video" sizes and bitrates.
Of course, if you throw enough bits at the problem, the differences between codecs start to disappear. I was unable to see much of a difference between H.264 and WebM using the same clips at triple the bitrate (1500 kbps). However, such a high bitrate for a low-res 640x360 video would be very uncommon. YouTube, for example, uses ~500 kbps for the 640x360 rendition of a video.

Source of lossless HD test clips used in this video, which were resized to 640x360 before encoding.

Presets versus quality in x264 encoding

2010-12-14T20:44:00.029-06:00

I'm scoping a project that will require re-encoding a large training video library into HTML5 and Flash-compatible formats. As of today, this means using H.264-based video for best compatability and quality (although WebM might become an option in a year or two).

The open source x264 is widely considered the state of the art in H.264 encoders. Given the large amount of source video we need to convert as part of the project, finding the optimal trade-off between encoding speed and quality with x264-based encoders (x264 itself, FFmpeg, MEencoder, HandBrake, etc.) is important.

So I created a 720p video comprised of several popular video test sequences concatenated together. All of these sequences are from lossless original sources, so we are not re-compressing the artifacts of another video codec. The sequences are designed to torture video codecs: scenes include splashing water, flames, slow pans, detailed backgrounds and fast motion. I did several two-pass 2500 kbps encodings using the x264 presets distributed with the x264 command line encoder (version 0.110.1820 fdcf2ae). Excepting the "ultrafast" preset, which does not use B-frames and was dropped from the charts as an extreme outlier, all of the presets created files that varied by less than 0.2% in bitrate.

Here is the actual encoding command used in the test:

x264 --preset {presetname} -B 2500 --ssim --pass {1|2} -o {output}.mp4 input.y4m

The mean luminance SSIM (as reported by x264) was used as the objective quality metric in my tests. Yes, I know about the weaknesses of using arithmetic means for video metrics, and the benefits of box-and-whisker plots for showing variance between frames. However, a single number is quite illustrative, especially since we are testing the same basic codec at the same bitrate with marginally differing tunings. This was a quick-and-dirty test. If I find the time to get avisynth working correctly on my Windows 7 x64 machine I will update the plots to include variance information.

Here's a quick and dirty chart (click to enlarge):

I was quite surprised that there was only a 0.75 dB difference in mean SSIM from the veryfast to placebo presets, despite placebo being 68 times slower than veryfast mode. I would have expected much more quality improvement for the CPU effort expended, given that placebo was producing just 1.5 fps on an eight-core machine. From a subjective standpoint, the results are indistinguishable to me, even going frame-by-frame through tough sections of the video.

Needless to say, all of my x264 encoding will now be done with medium preset or faster. Decoding a lossless source video for re-encoding became the bottleneck with medium presets or faster. If there is interest, please leave a comment, and I will find a hosting spot for the lossless, veryfast, and placebo versions of the video so others can compare, reproduce, or extend this simple test.

Did ND get any better in 2008?

2008-12-11T09:31:00.024-06:00

With all the talk about the potential firing of Charlie Weis (which ultimately didn't happen) I thought I would ask this question: did the Irish improve substantially between 2007 and 2008?

The most obvious way to answer this is to use the computer polls. I chose the Colley Matrix, which is part of the BCS formula, since it has a characteristic that makes comparing teams year-to-year straightforward. All teams in this computer poll are inherently normalized to a rating of 0.5, which makes comparisons year-to-year much more valid.

Here's what it looks like (pre-bowl games for both years):

Year	Ranking	Rating	Record	Strength of Schedule	Schedule Rank	Best win
2008	58	0.529	6-6	0.533	44	#45 NAVY
2007	90	0.375	3-9	0.604	10	#46 UCLA

First off, it is clear that the team was much better this year. The rating differential of 0.154 is the same as the differential between 2008 Ohio State and 2008 Cal - clearly a big step forward.

Secondly, the "weak schedule" of 2008 turned out to be not-so-weak: it was a lot tougher than the 86th-ranked "mighty SEC" schedule played by Alabama, for example. Tougher than 6 of the top 10 teams in 2008 in fact.

Yes, the team still looked inept at times. And the loss to Syracuse was frankly unforgivable. But there were also many bright spots in 2008 as well. Michael Floyd and Golden Tate are going to be one of the best WR tandems in the country next year. The defense improved. Claussen looked better. Special teams looked better, with even Brandon Walker finishing the year strong.

In the end, I'm glad Charlie is staying for another year. The last thing ND needs is to hit the reset button on the program without having locked up a great successor. Willingham was fired because Urban Meyer was on the market, and ND failed to land him, so we have Weis. There is no Urban Meyer out there this year. Until there is a can't-miss candidate available, and it is clear that he will actually take the ND job, Weis is the better option.

Network-focused analysis of the Windows Time Service

2008-03-10T12:18:00.013-05:00

Due to some recent posts on the comp.protocols.time.ntp newsgroup, I took it upon myself to investigate the behavior of the Windows Time Service a bit further using the Wireshark protocol analyzer.

It appears that in Windows XP, 2003, and Vista, the Windows Time Service (w32time) will by default always try to form a "symmetric active" association with configured NTP servers. This can be problematic with some time servers, violates the published RFC-1305 specification, and is not necessary. I could find no explanation on Microsoft's site for this behavior; I suspect it has something to do with interoperability with older Windows 2000 domain controllers that had very broken NTP.

However, there is a simple workaround. You can simply add ",0x8" to the end of any configured time server, and Windows will only use a client-mode association. For example, the command:
w32tm /configure /manualpeerlist:"0.pool.ntp.org,0x8 1.pool.ntp.org,0x8 2.pool.ntp.org,0x8" /syncfromflags:MANUAL /update

will configure your Windows machine to form client-mode associations with three different NTP Pool servers.
The minimum polling interval on all Windows machines except domain controllers is set to 1024 seconds by default. Windows domain controllers have a minimum poll interval of 64s.

This is reasonable as clients usually do not need extremely accurate time. However, quite a few servers that are not domain controllers do need to get accurate time offset and frequency synchonization quickly. You can configure "MinPollInterval" and "MaxPollInterval" through the registry or using Group Policy tools, as documented here.

Important note: never set the minimum poll value to less than 6 (which is 2⁶ = 64 seconds). You won't get better time synchronization, and will be abusing the servers you have configured. Many time server administrators have automated tools that block clients that poll too frequently.
Windows Time Service does follow sensible rules for "backing off" the polling interval, and adjusting the interval to network conditions. In my testing, a Windows Server 2003 domain controller began polling at 64 seconds, and then backed off to one poll every 1024 seconds within about 30 minutes. This is the same behavior as the reference ntpd implementation.

Also, in my tests, Windows Time Service did respond to unreachable servers sensibly, backing off the polling interval to 2¹⁵s. However, when a server became unreachable, it did increase polling in steps down to 2⁴s before reverting to 2¹⁵s. This rather strange polling pattern (15-9-9-8-7-6-5-4) continued until the server became reachable again. There have been quite a few problems in the past caused by NTP implementations that polled too frequently. Fortunatley, the Windows Time Service should not cause problems in this area, as an unreachable server results in an average of one poll every 2¹²s (about once an hour).

A cynic's view on the Sun-Network Appliance lawsuit

2007-11-12T08:57:00.001-06:00

I've been thinking about the Sun - NetApp lawsuit, which is an interesting case that highlights what is wrong with the Patent Office in the US. I don't necessarily think that all software patents are bad, and that innovation should be rewarded with temporary monopoly on a piece of technology.

However, this lawsuit shouldn't have happened, because at least one of the patents in question should probably not have been granted in the first place. David Hitz, founder of Network Appliance, has a blog in which he contends that Sun's ZFS violates patents held by NetApp for their WAFL. If that's truly the case, Sun should certainly be held liable, and should stop publishing ZFS as open source code. You can't give away what isn't yours.

The larger issue, in my opinion, is that it seems significant claims of the WAFL patent should never have been granted. There exists a significant amount of prior art in the use of a "tree of block pointers" to maintain logical consistency in data storage. Relational database management systems (Oracle, Microsoft SQL Server, IBM DB2, Sybase, PostgreSQL, etc.) have been using the same techniques for decades to maintain transaction-consistent indexes and relational data inside databases. WAFL may indeed be the first implementation of the idea where the "tree of blocks" points to files in a general-purpose file system. But in an RDBMS, the tree of blocks (tree of index pages) typically points to arbitrary row data in the database. What's the difference? Not much, bits are just bits. The application to a file system seems obvious to me (reasonably skilled in the art), meaning the patent should likely be challenged. Heck, not-so-innovative Microsoft was kicking the same ideas around back in the early 1990s as part of the WinFS file system for the "Chicago" project.

In fact, I personally designed the database for a document management system that used a "tree of blocks" to store and organize arbitrary file data in Microsoft SQL Server back in the late 1990s. This system had point-in-time recovery and look-up capability, based on valid time-stamps in the block pointers (folder and file tables with their indexes). I suppose you could call this a "snapshot" capability. The database took care of transaction logging, check-pointing, and referential integrity, all of which seem to be additional claims in the WAFL patent.

I have passable programming skills, I am not an algorithm design guru, and I had certainly never read the WAFL patents before implementing this "file system on a database". The basic ideas were widely known and used frequently in the database arena. I am not an intellectual property lawyer, but experience leads me to believe there isn't much innovation in the "always-consistent tree of blocks" described in WAFL patent. They devil may be in the details, I suppose - I have only reviewed the patent abstract at this point.

Still, in my opinion, the U.S. Patent Office, as currently constituted, is incapable of identifying true innovation. It grants far too many patents on obvious or derivative technology, especially in the software arena. If they can't get it right, even with the enormous resources at their disposal, they should probably not grant any software patents at all.

As a side note, this in-house document management system was never widely used, and the project was considered a failure. I believe this was largely the result of a cumbersome legacy ASP-based web front-end, though, not because of deficiencies in the storage engine.

time.windows.com fixed

2007-04-23T16:28:00.000-05:00

Well, it appears that time.windows.com is now fixed, after a few weeks of serving up invalid time. Presumably, the clocks on millions of Windows machines worldwide are now slowly drifting back into synchronization with the rest of humanity.

I find it rediculous that such a problem could go unnoticed and unfixed by Microsoft for so long, and that it took a Microsoft participant on a programmer's blog reading about it to track down and correct the issue.

U:\>w32tm /monitor /computers:time.windows.com,us.pool.ntp.org
time.windows.com [207.46.130.100]:
    NTP: +0.0541156s offset from local clock
        RefID: time-nw.nist.gov [131.107.1.10]
us.pool.ntp.org [66.91.129.70]:
    NTP: +0.0293621s offset from local clock
        RefID: bigben.ucsd.edu [132.239.1.6]

time.windows.com is broken... is your clock off too?

2007-04-13T15:08:00.000-05:00

It seems that time.windows.com is broken, reporting unsynchronized time off by two minutes or more. Why is this a big deal? Time.windows.com is the default Network Time Protocol server used by the Windows Time Service in Windows XP, 2003, and Vista systems. So there are literally millions of systems out there without an accurate source of internet time.

I have personally reported the issue to Microsoft, and Akamai as well (they seem to host the actual servers). But there has been no response from either for several days. Reports on the internet indicate that time.windows.com has been broken for at least a week!

Fortunately, it is easy to switch to a different time server. If your computer is part of a Windows domain at your workplace, it will get time from your domain controller by default, so you don't need to do anything. If your system is a domain controller, or is stand-alone, you should run these commands: C:\>w32tm /config /manualpeerlist:"us.pool.ntp.org,0x8" /syncfromflags:MANUAL /update
C:\>w32tm /resync /rediscover

Note these commands do not work on Windows 2000. For that the command would be:
C:\>net time /SETSNTP:us.pool.ntp.org

What is us.pool.ntp.org? It is the United States address for the global NTP Pool Project. You can substitute your own two-letter country code for the "us" portion if you are not in the continental United States. If you're in London, for example, use uk.pool.ntp.org.

The ",0x8" after the time server name tells Windows to use a client-mode association with the time server. This isn't strictly necessary, but the proper way to configure an NTP client talking to an NTP server. If you don't use it, Windows checks the time exactly once per hour, rather than adjusting its time-checking interval automatically based on clock performance and network conditions.

One final note, do not use popular "stratum-1" time servers to synchronize your client. These systems are typically run by national standards laboratories (an example would be time-a.nist.gov). These time servers are quite overloaded, and Windows systems cannot use the increased accuracy they provide anyway (the Windows Time Service is only accurate to about 16ms). The NTP Pool Project was started for the specific purpose of reducing the load on the Internet's stratum-1 time-keepers.

Linux plunge not working out so well

2007-04-01T15:31:00.000-05:00

So I don't think Linux is ready for laptops. Well, my laptop anyway.

The first problem I encountered was with screen resolution. I didn't have an option for the "native" 1280x800 widescreen resolution of my Dell 700m. After digging through the Ubuntu support forums, I discovered that I had to install a small utility called 915resolution. It was a minor pain to track this down, as there were several contradictory sets of instructions found with Google, but running this command:
sudo apt-get install 915resolution
and restarting seemed to fix things.

My next problem was with WiFi. Ubuntu's network management applet didn't show any available wireless networks, despite the fact that I know there are dozens nearby my home. Reboot into Windows, do some more browsing, and discover some diagnostic tests to run. Boot back into Ubunutu. It appears that command-line tools can see wireless networks nearby, but Ubuntu's GUI is broken and doesn't list them. I could deal with having to run a few commands to connect, but...

It also seems that Ubuntu does not support Wifi Protected Access 2 (WPA2) wireless access points out of the box. This is a much bigger deal, as my home network is all WPA2 WiFi, and I will not "downgrade" to any version of WEP, as it is woefully insecure.

So I boot back into Windows, and do a few more hours of research. There are WPA tools for Debian-derived Linux systems, and some folks have gotten them to work. But I was really unwilling to go down this route, as the documented procedures were pages long, and involved running a command line utility to generate a password hash for each new network I wanted to use. Not exactly useful for someone who needs to do work on the go.

Finally, I looked into getting support for my Sprint Mobile Broadband Card, which provides about 1 Mbps download speed just about anywhere. This little device is my lifeline for work. From what I read on the net (again scattered over dozens of contradictory sites), there is almost no driver support at all for these mobile Wireless cards in Linux. To get something working, I would have to modify some available driver for another device and compile it into my kernel.

So I gave up. I have a family, and a job, and I just wanted to get some work done. Right now, at least, Ubuntu doesn't have enough mobile device support for my needs.

And I know all the Linux fanboys out there will call me an 1d10t n00b, and blame the hardware manufacturers for not releasing good open-source drivers. But you know what? I don't care. Ubuntu failed me. Going mobile with Windows XP is light-years easier by comparison, and I'm not going to switch to something that requires so much manual configuration each time I want to work on the road.

Maybe I'll try a MacBook instead.

Taking the Linux plunge

2007-03-23T16:49:00.000-05:00

So I've got a decade or so of Windows-based network administration experience. However, my formal computer science education was rooted in Unix back in the early 1990s. We used Sun workstations exclusively back then, and almost all programming was in C or Scheme. I even wrote a simple C compiler.

But my first job was at a Netware 3.X shop, which we transitioned to Windows NT 3.51, and I've been managing large, multi-site, but mostly-Windows networks ever since. (I've also done a lot of DBA and security work too). So I'm not a neophyte when it comes to IT, but I don't have much Linux experience, as none of the jobs or projects I've worked on used Linux.

I finally decided to install Linux and actually try to use the thing regularly. I've done "toy installs" of various Linux and BSD flavors over the years, mostly in in Virtual machines. But this was my first real go at using Linux regularly.

I picked Ubuntu 6.10 as my distribution, based mostly on reputation as the simplest Linux to get working.

But first, I had to partition my hard disk on my Dell 700m so I could dual-boot. No problem; I've used PartitionMagic before. But QTpartEd from the bootable Knoppix CD can also shrink and existing NTFS partition, so I used that, and it worked like a champ: 40 GB for my existing Windows XP Pro installation, and 20 GB of empty space for Linux.

Not exactly "my mom can do it" easy, but not the sort of thing that's necessary unless you want to dual-boot.

After downloading Ubuntu 6.10 installer and burning it to CD, I fired it up in my road-weary Dell. A nice GUI installer comes up, and asks a few sensible questions. Then it asks where I want to install. But the default option is to erase the entire disk and devote it to Ubuntu!

Now, not even Microsoft has the gall to default to "erase everything else" the Windows installer. But choosing the "use largest free space" option was easy enough. So I'll forgive this, and I'm sure there would have been lots of warnings to prevent me from killing my Windows partition if I had chosen the default option.

Now click through a few more sensible screens, wait 20 minutes or so for files to copy, and then a reboot. Ubuntu comes up in all its earthy (that is, very very brown) glory.

All in all, a simple install. Simpler than Windows XP, in fact, and about the same as Vista.

But now the real fun begins, as I try to use this thing regularly.

Cooperative backup and archiving

2006-08-22T17:57:00.001-05:00

After testing my company's shiny, new 100 Mbps internet service from Cogent, I was struck with an idea. Plenty of companies out there offer internet-based backup service, but most of these are priced per-gigabyte-per-month and are prohibitively expensive. Especially when you consider that our full backup sets are typically 600 GB in compressed form, and we of course want to store daily incrementals for several months.

My idea: find another Cogent subscriber, and enter into a mutual backup agreement. That is, we buy a storage server to sit on their site, and they buy one to sit on ours. We can then exchange backup traffic, eliminating the need to shuffle tapes and send them off-site. Many large enterprises already do this with SAN hardware replication; however this would be a budget "roll your own" solution.

There exist several open-source distributed storage systems that might help, however, none seem ready for prime-time yet. I feel quite a lot could be accomplished with judicious use of native backup tools and open-source encryption software.

Ideally, we could use something like rsync or rdiff-backup to send only the changed data each night. However, since the backup server will be at an untrusted site, all data must be encrypted before it leaves our network, and no decryption keys can exist at the untrusted backup site. Since all good encryption methods produce entirely different byte streams with each encryption, tools like rsync won't gain us anything.

My current thoughts are to use native backup tools to create a local file-based backup, and then use GnuPG or a similar tool to compress and encrypt full backup files for transmission via FTP. With weekly full backups and daily incrementals, we'll be transmitting a lot more data than we would with rsync, but a 100 Mbps connection could make it workable (~20 hours for a full backup over the weekend, and much shorter incremental times).

One thing that could drastically reduce the amount of data that needs transit is some form of single-instance storage. Using native tools, we'd be storing dozens of copies of many binary files (OS and application files). However, I can't see how this could be made to work when encryption has to happen before file transmission.

Any thoughts?

Windows 2003 SP1 has a "real" NTP service

2006-05-17T16:53:00.000-05:00

It appears Microsoft has finally gotten around to making the Windows Time service into a "real" implementation of the network time protocol in Windows 2003 SP1 and later. The half-baked Simple NTP system of earlier Windows versions kept poor time (it would often drift by as much as a full second) and had troublesome interoperability characteristics, reporting the incorrect stratum amongst other issues.

My week-long tests indicate the updated service keeps time to within about 15 ms of a "real" NTP server configured with the same set of Stratum-1 sources. This is about the resolution of the Windows system interrupt timer (1/64 second). The updated service also seems to report the correct stratum, which server it has selected for synchronization, and a seemingly correct resolution value of 2^(-6) seconds. Here is a chart showing the performance relative to a server running ntpd.

Of course, diagnostics are still terrible in this updated version, so my only recourse is to monitor the new Windows Time Service with the tools available on a system running the standard ntpd.

Comparison of compression programs

2004-06-08T11:12:00.001-05:00

I have done a reasonably detailed test of popular compression programs for archiving files. The results were a bit surprising.