Alex Hopmann’s Blog

So Windows (like most OSes) maintains a number of caches where it keeps in RAM frequently accessed disk contents (as well as a few other things). Normally these are not adjustable- it just uses extra free memory for this to improve performance, and the minute you actually need the memory, it is freed up and available to a real program.

Hyper-V puts a little twist in this. When you create or start up a new VM, it needs the full memory as hard-allocated memory to that VM and apparently it doesn’t force a cache clear if there isn’t enough memory available. We are building a bunch of stuff on 48GB servers and initially thought we would easily be able to put 3x 15GB VMs on them- the remaining 3GB would be plenty for the OS that basically wasn’t doing anything. But at times we have found the Cached memory is as high as 20GB (after a bunch of big file copies, usually of VHD images), and the machine doesn’t have enough free RAM to boot the target VMs.

Any thoughts on how we can clear this? Dynamic memory coming in future releases should fix this issue, but for now it would be great if we could just run a cache clear/flush right before VM start time.

For someone with such crappy handwriting its surprising that I’ve always been so interested in using it as an interface to computers. Most of the time I don’t care at all about voice user interfaces- it seems like a giant pain to talk to my computer (although some of the phone scenarios are very interesting). But let me put it this way- I have one of the Newtons from the first day the first model went on sale.

Of course the reality of handwriting interfaces to computers has been pretty much a continual disappointment. I was hopeful about the whole idea of the TabletPC since it seemed like a much better way to both have a device for reading plus have a computer in meetings without the effect of everyone sitting around in a room with a screen up in front of their face. Somehow having a tablet sitting on the table feels less rude to me (although I suppose it can easily be just as bad).

Unfortunatelly the form factors of early Tablet PCs and the user experience of the handwriting input has been bad enough so far that it hasn’t been worth using. Ever the hopeful type when I needed a new laptop at work I opted for a Tablet and got the HP 2730. Now, the 2730 isn’t great- I’ve got a long list of complaints and let’s just say that it isn’t really competing with a Kindle as a reading device.

But the great news is I’ve been running Win7 lately and I’ll say that the most noticeable breakthrough in Win7 for me has been how well the handwriting user experience works. It still isn’t perfect, but the current flaws are more like minor oversights than huge flaws. For example, when you go to click on the button in the top left of Office apps, the handwriting panel is right there and likes to pop out. But overall the experience is 1000% better and so far has finally gotten good enough to be solidly useful. I wouldn’t want to write a long document using the handwriting, but mostly because I can type about 5x faster. But for responding to brief emails and IM messages, and general interaction with the system it works great, the handwriting user experience seems smooth (I even like the font they use as it translates your handwriting) and so far I’m still using it and expect to keep using it.

Tom’s Hardware has a write up on BPOS (the Business Productivity Online Suite) that includes the SharePoint Online stuff that I’m working on.

I’ve been needing to finally get around to posting this. I’m back at Microsoft, working on the Sharepoint team to help deliver our grid (hosted cloud) service.

Hopefully I’ll have lots more fun stuff to say about this soon. But I need to get this post out because I’ve been meaning to for weeks and until I actually get around to hitting the “Publish” button all kinds of other things I’ve wanted to post are blocked up behind it. A classic case of when your thought process behind a blog post becomes too complicated for the medium.

Yesterday I wrote about problems installing Vista SP1 on this machine. Running the “download and run it yourself” installer worked fine with one little glitch. The issue was that I first downloaded the SP1 setup that only contained 5 languages, thinking “hey, I only really actually care about English”. The catch is that it won’t run if you have any languages other than those 5 installed. I had previously installed a ton of languages since they were just check boxes in Windows Update.

Uninstalling languages turned out to be a nightmare. First of all, you can’t uninstall from “Programs and Features”, you have to go into “Regional and Language Options”, go into the “Keyboards and Languages” tab, click a button, click another button. Then most of my attempts to remove languages would fail (after sitting doing something to my computer for what seemed like 30 minutes), PLUS it requires a reboot after each one (even though I’m uninstalling languages I’ve never ever used).

So the solution ended up being downloading the “all languages” version of the Vista SP1 package. The 20 minutes the download took was way faster than uninstalling the languages. Upgrade felt like it took a long time- I could be wrong but it feels slower than initial setup (which does make some sense since initial setup can just write an image down on your disk while upgrade is presumably doing more real work).

All is well now and so far the machine is running smoothly.

I’m having a problem installing Vista SP1 on my main workstation. Microsoft Update keeps throwing a Code 80070570 error which has no real description. The best hint I can find searching the net is that it might be related to some disk issues, but running a CHKDSK to fix errors didn’t make the problem go away.

My next attempt is to download the standalone upgrade- its possible the problem I’m experiencing is with Microsoft Update rather than the service pack itself.

Just saw a post about Microsoft Velocity, Microsoft’s answer to memcache. I’m looking forward to checking this out soon- we have had a ton of success using memcache on the LAMP platform and it was a missing piece in the .NET world. I understand why it was a bit hard to see its importance in that environment- Unlike PHP, .NET can persist things in memory in between requests. PHP really needs memcache quickly since you can’t really save anything from request to request so you go to the database very quickly. But as your system grows Memcache fills an even more important role because of how easily you can scale out adding more caching servers.

Memcache (at least the standard one) has a few problem scenarios all revolving around how it so easily shares the load between servers. The coolest thing about memcache is how simple of a mechanism it is- no complex configuration to tune (there are a few things to tune), maintain, debug. You simply configure N memcache servers on all your web server, and when you save or retrieve an object, it takes the ‘key’ (an arbitrary string), hashes it and uses the hash to pick one of the memcache servers (in effect it does H(key)%N where H is the hashing function and N is the number of servers). In effeect you automatically get a smooth distribution of your keys across your caching servers.

However, if you add or remove memcache servers from the array, it changes the hash, so all of the sudden your keys are on different servers. Now, if your site is under low load, this isn’t that big a deal- you just dumped your whole cache and it will build back up and be fine. But if you are running memcache because you really need it, your site just went down as all the web-servers just started pounding directly on the database with every request. Right now DeepRockDrive has a pretty unique situation where we get huge spikes of traffic (that are mostly predictable- showtime) during which the memcache servers really save our bacon, and most of the time we can clear them out more or less safely, but most big sites are going to have a more consistent traffic pattern and would have a harder time with this.

This also means that if you a memcache server goes down you can’t just pull it out of your configuration (at least under normal load). You really really need to replace it. The easiest (although resource intensive) way to do this is to just have a hot-spare server or two in your track. If one of your memcache servers goes down, you map that spare to the same IP and bring it up. You just lost a portion of your cache (10% if you have 10 servers, etc..), but its way better than losing the whole thing. A more complicated setup would be to run multiple instances of memcache on every machine. So if you have 10 memcache machines, you run each with 3 IPs and it looks like your whole array is 30 “servers”. If one goes down, you bring those 3 IPs up on 3 of the other machines distributing some of its load to those machines.

We hadn’t had a chance to fully work out the scalability of the standard memcached running with multiple instances on the same box so far. We have played with it some and on our 8-core boxes even with the right threading libraries we haven’t gotten close to maxing it out with a single instance of memcache. It looks like there may be some I/O limits but I can’t be sure about what is actually going on, still the notion of running multiple instances on the same box seems like a fairly reasonable one for scalability and these fail-over flexibility cases.

The other tricky issue is that memcache gives you a balanced distribution of keys but does not necessarily give you a balanced distribution of access. Lets say you had some runtime configuration information that you wanted to persist on your site. The easy thing to do would be to save it in a key called ‘config’ and just retrieve that key on every request.

What you have accomplished here is to just introduce a nice hard-scalability limit into your system. Memcache isn’t actually that much faster than MySql is for basic queries. If MySql can cache the query well (as it would be able to for a query on a simple table that just gets hit over and over), the performance of the two will be pretty similar. Where memcache shines is that because of how its keys and the hashing thing work, it can transparently distribute that load over the multiple servers. So the person building the app with the ‘config’ key will have something that looks great as its small and on one caching server and then when they try to apply it to a high load site with multiple servers. All the traffic still just goes to the one memcache server (since its one key that gets hashed the same every time) and they will typically be stumped why the performance isn’t better.

The way around this is to generate keys that look something like-

‘config-’.rand(0,9)

(php syntax)

At first this is counter-intuitive. I’m storing the same thing in 10 keys? That means that when they expire I’m going to have to go back and do 10x the initial loading of this object (whether from the database or config files or whatnot). However, at the cost of a very small # of those database queries (they only happen once every 5 or 60 minutes right), I’m spreading my keys out across my memcache array and the result is that the load gets spread smoothly across my whole array. I can even do a couple of slick extra things like every time I refresh the config data write to all 10 keys at once, resulting in no extra load on the database (except for a race condition I’ll cover in a future post) and just a small amount of occasional load on one web server.

I started this post mentioning Microsoft Velocity and then went into memcache- looping back to Velocity, in typical Microsoft fashion it looks like its a much more complicated solution, but it also automatically deals with some of the above issues. As far as I can tell from a few architecture diagrams the servers maintain knowledge of a cluster (memcache servers have no idea about each other) and I’d assume they automatically deal with some of the fail-over and “add a server” cases. It also has a more explicit concurrency model- although memcache supports building things with much of the same concurrency behavior you need to manage it a bunch more yourself.

Looking forward to checking out velocity more later. In particular I’m interested in when the protocol to talk to the servers will be published and whether there will be support for PHP/Python clients talking to these servers.

I mentioned last week that I was going to have a hard time playing with Mesh because it won’t run with UAC disabled. First of all, Microsoft did post an explanation of why the current version requires UAC. I understand the explanation, but it is one of the typical cases where teams have to build on this internal technology and that one and this other one, none of which are ready for prime-time and all of which aren’t necessarily focused on actually providing user value…

In any case, I did install it on two XP machines. I’m about to shut it off. I noticed one of them was running a bit slow lately and checked the task manager. Mesh is using 490MB of memory, and has consumed 9 hours of CPU over the past couple of days. And in case it wasn’t clear, I haven’t really been asking it to DO anything…

I’m also mildly annoyed that the process is named “Moe.exe” so it took a small amount of digging to figure out what it was that was consuming all my system resources. They also have a second process called MoeMonitor.exe that somehow has consumed over 1.25hours CPU and 33mb RAM. It appears to be responsible for the task-bar icon.

Microsoft is releasing the sources for the .NET frameworks including most of the base libraries, XML, and even stuff like WPF. This is great- it will make it a ton easier to debug things, understand performance implications and more.

There are some people complaining that they aren’t open sourcing this stuff, but I think that is unfair. What is great here is that they have realized that they can still retain the rights to the IP while providing these sources to developers to make the platform better to build on.

Dare posted links to a Back to the Future Spoof that was used in Bob Muglia’s TechEd keynote. Great stuff. I’m touched that Bob managed to include “web store” in the great list of failed integrated storage attempts. Ahhh, good times.

I also love the “get me all the pictures of me with the CEO at the company picnic for the past 5 years”- a good semi-inside joke. I knew WinFS was in trouble when this was the most common example of why you really needed to be able to do joins on semi-structured data like documents, photos and contacts. It was a classic case of “so we have this great technology that does this thing X. Hmmm, why would anyone actually care about doing X. Lets construct a scenario that needs it!”

To be clear, its not that WinFS couldn’t do lots of more basic things that people cared about. However, other, less complex technologies could also do those other things so the team had to keep constructing odd scenarios that those alternative technologies couldn’t accomplish.