Adam Fields (work stuff)

Some tips on getting started with Vagrant and Chef

Mon, 20 May 2013 11:44:45 -0400

We have recently switched from a manually configured development environment to a nearly fully automated one using Vagrant, Chef, and a few other tools. With this transition, we’ve moved to an environment where data on the dev boxes is considered disposable and only what’s checked into the SCM is “real”. This is where we’ve always wanted to be, but without the ability to easily rebuild the dev environment from scratch, it’s hard to internalize this behavior pattern. Cruft builds up in the dev boxes, and it starts to slowly drift out of sync unless a lot of manual intervention is applied. Rebuilding the VM is not something you’ll want to do (or have to do) very frequently, but it’s a good feeling to know that you can with very little pain, if it’s needed. This still doesn’t completely eliminate development cruft, but the hope is that it can significantly cut down on it. If you’re worried about running vagrant destroy, that’s a hint that you might have done something wrong.

This post covers our ruby dev environment on Mac hosts. For other platforms it will be slightly different, but the principles are the same - use vagrant to manage your underlying machine configuration, automate as much of the rest as you can, and keep all important data resident off your dev boxes, checked into scm or using another redundant remote mechanism. As someone who’s been doing unix system administration as a sideline to development for about 20 years, I’m really impressed at the extent to which the automation tools have improved in the past few years. I did not have any direct experience with configuring chef when I started this, though I’ve been familiar with the concepts and have used other automated configuration and deployment mechanisms.

The configuration took a few days of solid effort to get up and working to the point where it was usable, and then a few more days to work out the kinks and figure out our preferences. While working through it, it was pretty easy to identify what the next step was given what wasn’t working. Work your way clockwise around the diagram - get vagrant up and running, then do base installs, then work on getting your source tree in place, then get it working, then import dev data. Instead of distributing ssh keys to the individual VMs, we use agent forwarding to allow each VM to use the keys from its host. The final config consists of only a handful of files. Depending on the complexity of your environment, you may need some additional scripts:

The Vagrantfile which determines the vagrant and chef configuration.
A Cheffile which specifies to librarian-chef which cookbooks to get.
A custom shell script which populates ssh config files, sets up user and database permissions, and performs project checkouts with the appropriate rvm environment.
A shell script to run on the host to add the host’s ssh keys to ssh-agent (probably automated by launchd).
A custom /etc/hosts file to insert into the VM config.

These files are checked into SCM, so bringing up a new dev box is only a few steps (all but the last step only ever need to be done once per machine):

Install virtualbox: https://www.virtualbox.org/wiki/Downloads
Install vagrant: http://downloads.vagrantup.com
Make sure your key is on the server for the dev config checkout.
Check out the dev config repository.
Run the ssh-add script to bring your keys into the ssh-agent.
Install librarian-chef using any local ruby (gem install librarian-chef)
Install the librarian chef vagrant plugin, so vagrant will automatically fetch your cookbooks (vagrant plugin install vagrant-librarian-chef)
Set up any local path location and config variable overrides in ~/.vagrant.d/Vagrantfile (optional)
Bring up your dev environment with vagrant up.

We use vagrant synced folders to mount selected folders in the host’s Dropbox folder and the separate project tree where checkouts are placed, to share these files between the VM and the host, both for letting developers use their own Mac-native environment to edit files in the source tree and for importing data via dropbox (Rather than checking them into scm, our database test packs are distributed via dropbox, and the provisioning script looks for them there to do the initial import.). Chef handles getting the proper ruby versions installed, and then bundler ensures that the gems needed for each project are present. We have used bundle package and checked in our vendor/cache folder, which greatly reduces the amount of time it takes to install gems.

With this configuration, the VM has its own copies of the databases and its own source trees. The databases and local configuration files (with some critical exceptions that are always overwritten at provisioning time) are wiped out in the case of a vagrant destroy, but otherwise preserved. The source trees are always synced with the host, so they will survive the destruction of the VM, but they can of course be manually wiped if needed.

Some random notes:

There was some complexity in having the provisioning shell scripts running as the vagrant user (instead of the default, which is root), with access to the ssh keys via the ssh-agent forwarding. The best solution I found was to make a separate script file, then call it with sudo -u from the provisioner. All of the configuration that needs to be done as the vagrant user (rvm setup, bundler installs, scm checkouts, environment config etc…) goes into that script.
The default VM runs with very little RAM and only one CPU, which makes compiling (especially new rubies under rvm) extremely slow. You’ll want to bump this up.
There’s a balancing act between what should be configured by chef and what’s easier to just do with shell commands. Most of the simple configuration settings and individual packages are easier to do with the command line.
There seems to be an intermittent issue where a suspended VM will be unable to pick up a DHCP ip address upon resuming, which causes it to hang indefinitely. An interim solution (found in this post) to this seems to be to execute VBoxManage guestcontrol <vagrant machine id> exec “/usr/bin/sudo” —username vagrant —password vagrant —verbose —wait-stdout dhclient from the host.
I find it odd that there’s no way to specify local Vagrantfile changes that take precedence over the project files. This means you either have to design your Vagrant config in such a way that you’re setting variables which can be overridden locally, or people have to edit the project Vagrantfile directly (and risk accidentally checking in their customizations). This seems likely to cause problems as different developers are going to want to use machines with different amounts of memory. We’ve temporarily worked around this by using global config variables and only setting them if they’re not already set.
Vagrant uses its own config file for controlling chef commands in a way that differs from chef’s own syntax, meaning that chef configs can’t be reused outside vagrant. I haven’t yet found if there’s a way to import generic reusable chef configs directly.

Our Vagrantfile (with some custom config removed) looks like this:

Setting this process up has definitely been some work, but I think it’s worth it once everybody gets set up. There are a lot of moving parts, and a lot of individual edge case problems to solve. But the nice thing about this is that once you do solve each one, those changes can be globally applied to everyone with little extra efforts. We’ve worked our way through a number of issues in a very short time - I’d love feedback on this from experienced vagrant users.

Why I'm not buying Bitcoins

Mon, 08 Apr 2013 14:28:19 -0400

I’m not an economist, but Bitcoin seems pretty scary to me right now. It seems attractive as a decentralized currency, but there are a few properties that are worrisome.

The value (exchange rate) has increased in the past few months past the point of “convenience money” for most. At an exchange rate of somewhere between $10 and $30, there are a large number of people who will buy some without fear that they’ll lose their money, and more importantly, at that level, you have a large number of buyers who bought in early enough to sell at slightly higher rates and still make a comfortable profit. Buying at $30 and selling at $170 (where we are now) is not a bad run by any measure. People are buying in now, but there’s a limited number of bitcoins in circulation and mining can’t keep up with the demand, so people must be selling. Who’s selling? - it’s the people who bought in earlier and are now taking profit on their earlier purchases. Flash forward a bit and imagine that the exchange rate has reached $500. If you want to buy then, you’ll be doing so from someone who bought in earlier and is looking to make a profit. That would be the people who are selling now plus the people who are buying in now. Now flip it around and say you want to sell at $500. Who’s buying? At that point, maybe the media frenzy has died down, or maybe there’s a fresh influx of people who think it’s going to go to $5000. Maybe it will. With no central clearinghouse, there’s no guarantee of being able to sell at any given price, and it doesn’t seem to me like this scenario is set up for long term success. At somewhere in this range and above, you’re talking real money for a lot of people, and the casual speculators will disappear.

Now, let’s look at the transaction rate. MtGox is the largest exchange, by an order of magnitude. As of the time of this writing, over the past 30 days, they have done 2172994 transactions, which is about .83 transactions per second. For a global currency whose market cap is measured at over a billion dollars, this is an astonishingly low number of transactions. (Let’s put that in perspective a bit - so far just today, Google stock was traded 1,990,139 times. Google’s stock price is determined by roughly 30 times as many transactions as all of Bitcoin’s.) Demand is high for now, but liquidity is low, and when it comes time for a lot of people to want to extract some value, there simply aren’t going to be a lot of interested buyers. Most of the previous buyers will have gotten in at near or above the current trading price and so will be waiting for it to continue to rise, a high enough entry price will chase away new buyers who can’t afford to lose their investment, and any dips will keep away the marginally interested people who were only showing up because of the media attention.

I expect the exchange rate to continue to rise for a while, but there’s really no way to know how high it will go, and I think eventually the trap will close on everyone all at once.

I’d love to hear arguments why I’m wrong about this, but I think the time to buy in has already passed.

How I do project planning (and why OmniPlan doesn't help)

Sat, 30 Mar 2013 11:24:16 -0400

I do a lot of medium-term software development project planning (typically 6-18 months). Here are the steps I use for doing project scope estimating (which is a distinctly different thing from tracking progress against the initial plan). After figuring out what rough timeframe the project should be targeted for, we:

Figure out what the tasks are and break them up into logical groupings.
Chain dependencies for tasks that cannot be done simultaneously.
Estimate how long they’ll take if one developer does them, setting the amount of effort. This is usually done on the scale of days per task, and any task that takes less than a day should be grouped into a larger bunch that can be estimated on the scale of days. We’re not micromanaging here, just trying to get a sense of the effort involved.
Fix the effort for all tasks, then add developers (with assigned salaries or hourly rates) to each bundle of tasks to bring the total time in under the deadline.
Be very aware that this is just an estimate, that adding developers to tasks doesn’t tend to strictly reduce the time by the number of developers assigned, and that a certain amount of padding is also needed for communication and integration. And do it anyway, because a wrong plan to measure against is better than no plan.
Resolve allocation conflicts.
Rationalize by hand any inconsistencies that don’t make sense (see step 6), and re-examine the long-term project timeframes (figure out if some tasks should be bumped to subsequent phases).
Answer the single most important question in project estimation - “given a time range, how much is this project or sub-task grouping going to cost me?”. This answer is critical - it determines the proposed budget for the project, and helps determine if a part of the project is worth doing.

I generally love OmniGroup software and I throw money at them at every opportunity I get, but OmniPlan has been extremely frustrating for this process. It is a great tool for steps 1-3, and it has turned out to be completely useless for steps 4-8. In both major revisions, my every attempt to do steps 4-7 has ended in frustration and a tangle of incoherent and incomprehensible unresolvable conflicts with no useful information about how to fix them. Theoretically, it should be possible to assign a bunch of developers to a group representing a class (senior developers, system administrators, etc…), assign the class to a task, and then get a free developer from the pool actually assigned to do the task. In practice, this has resulted in the necessity to assign far more developers in the group than should be necessary, with each developer only assigned to less than 20% of their capacity. The problem seemed to center around they way tasks were assigned to developers within a group, resulting in some of them being overloaded and dragging the schedules for everyone else forward, but trying to figure out what actually messed this up was an exercise in futility, and every time we’ve just given up and ended up estimating by hand.

Step 8 - budgeting cross-referenced by timeframe and task grouping - should be straightforward, but seems to be simply not possible. All I want to do is select a time range and then get a burn rate for the tasks included in that time based on their assigned resources. There’s a total task cost listed, but no way to tally it for just a specific time range without adding rollup tasks, which complicates the earlier tasks even more. For software I’ve paid hundreds of dollars for (and by far their most expensive), this is a disappointment. Numerous rounds back and forth with support yielded no progress before I gave up in frustration.

I’d love to hear any suggestions you might have for better ways to do this process.

(Update: it seems that they’ve released an update a few days ago which may help alleviate some of the pain in determining why tasks were scheduled late, but I don’t think that solves the fundamental “allocate these tasks to any developer in this group” scheduling problem.)

Changing ulimit successfully for databases on linux

Mon, 18 Mar 2013 14:36:00 -0400

Databases are a pain, but one thing seems to bite more often than anything else, especially at scale: per-user limits. OS creators have thus far not seemed to have caught on and made these settings either easy to set or easy to make persistent. Typically, when you run a production database, that’s going to be the only thing running on the machine, so it’s okay if it uses all available resources, and the default multi-user settings designed to protect users from each other make no sense. (Protecting yourself from runaway processes is another story, but generally speaking, you want to make your entire machine’s resources available to your database, otherwise what’s the point?)

This post is aimed at CentOS/Amazon Linux. If you use another distro, your steps will vary, but may be similar.

Here’s what I had to do to make this work (I’ll use the mongodb example):

1) Add lines at the end of /etc/security/limits.conf to increase the system limit for open files and number of processes, getting badly bitten by the apparently undocumented feature that newer kernels require you to specify nproc limits user-by-user and wildcards no longer work. You may also need to add the root user here - it’s unclear.

2) Modify /etc/init.d/mongod to set the runtime limits before the server is started (I’ve included the whole init.d script here for reference, in case you don’t have one):

3) Start up the server, and verify that this worked with this extremely helpful script.

Ridiculously useful bash snippet to return actual ulimit values for a given process

Fri, 08 Mar 2013 13:07:00 -0500

From the MongoDB docs comes this very useful bash function. In a shell, do: Then you can get the ulimit values of a running process with e.g.:

return-limits mongod

Evernote, you really messed this one up, and you're making it worse.

Sat, 02 Mar 2013 20:05:30 -0500

So… Evernote was hacked. Yay. Except that their “recovery” procedure is seriously, significantly, embarrassingly flawed. I’m flabbergasted.

The procedure is as follows:

Log in with your old password.
Change your password to a new one.

Yes, that’s right. All you need to change your password is your old, compromised password, and there’s no other verification required. No email verification, no secret questions, no nothing.

Evernote, I don’t think you understand how serious this is. This needs to change, immediately.

(If you have an evernote account, I advise you to go change your password NOW. Before someone else does.)

Security implications of oauth and leveraged access

Sat, 02 Feb 2013 11:12:00 -0500

So - twitter was hacked this week. Which happens to the best of us, and is going to happen more often in the future. They invalidated the passwords for 250k user accounts (though anecdotal reports would seem to indicate that it’s more than that). But here’s the strange thing - even though my password was invalidated and inactive, the oauth tokens for apps I’d approved access for still worked. And continued to work even after I’d changed my password. If they thought it was possible that my account was compromised, why weren’t these tokens deactivated, even temporarily?

Granted, most of these access tokens are for mobile apps, or for using my twitter credentials to post on someone else’s site instead of drawing from my twitter feed, or for enabling other apps to post _to_ my twitter stream on my behalf, but some of them are reading my stream, including direct messages, and someone with my password could have potentially spread the damage around to other services (adding significant hassle to the cleanup), and also added more without my knowledge. (It’s possible that they deactivated more recent ones.) They did ask me to review them after resetting my password, but it was entirely optional, and a less savvy user may not have known what to look for.

This seems like a serious and surprising oversight. I think most technical people barely understand the implications of authorization delegation, let alone everyone else. It seems like it should have been a reasonable precaution to at least turn all of those off until my password was changed.

The difference between task and list managers

Thu, 20 Dec 2012 13:24:41 -0500

There seems to be a lot of confusion about the difference between task managers, to-do lists, and checklists, and why you’d want different apps for them. They’re three very different working patterns with different goals, and they require different UIs to be successful.

Disclaimer: I don’t make any presumptions of being an expert user of these apps. I am not a “productivity guru”. I am probably missing some very useful features of some of these apps because they haven’t surfaced them well enough to be obvious, and I resist modifying my natural workflow to do things the way apps want me to without a good reason. I have work to do. I’m productive because I do my work instead of fiddling about with settings, and I primarily use these tools to help me not forget important details, not to tell me what I should be working on. Everyone needs to find their own style, and may find different UIs more or less intuitive. This is some discussion about what I use now and how well it works for me.

A note on Sync: task managers of all flavors are useless if you don’t have them with you when you need them. I consider sync to be critical for everything except the most basic and narrow use cases. All of the apps listed here have iOS and Mac versions that sync, except where noted.

There are many different classes of tasks. Some examples from my daily life:

things you have to do right away
things you have to do later at a specified time
things you have to do later but you don’t know when
things you have to do later but it doesn’t really matter when
things you have to do after some other thing is done
things you have to do frequently but on an irregular schedule
things you have to do every day
ad-hoc groups of things you have to do together
regular groups of things you have to do together often

Task managers, as a class of applications, aim to keep your entire perspective across many different contexts. They can handle complicated metadata like start and due dates (and reminders), dependencies, hierarchies, priorities, task status, task states, estimated times, and notes.

My primary store of tasks for the first five of these is OmniFocus. (I’m told Things is very similar. Having bought OmniFocus, I don’t see a good reason to spend a lot of money on Things. If someone wants to make the case that Things is significantly better than OF in some way, I’m all ears.) I don’t do GTD, and while OF is fairly optimized for the GTD use cases (if you want to do GTD, it’ll work well for that), it is a flexible task management system with a lot of applicable metadata. It’s always seemed too much work to me to set up and manage contexts, and I don’t strictly use OF to tell me what to work on next. I use it primarily as a store of all of the tasks I have to do at some point, things I don’t want to forget, and projects I’ll probably never get to. I use the project hierarchy extensively, as well as due dates for the current project. It is not representative of the actual list of tasks I’m working on next, except for the OF project called “Do these next”. (At one point, “Do these next” was a context, but I found it easier to just make it a separate project.) My job is such that pretty much as soon as I’ve put something on the list to do next, there are going to be five or six other things that pop up that need to be done first. I use OF largely to keep myself from getting too far off track over time. I try to be judicious about actually killing tasks that are on my list that I never really intend to do, and moving them to inactive projects. I frequently come up against tasks that would be nice to do, but never end up being important enough to take priority over more immediate concerns. I’d say most of those tasks end up having about a 20% chance of ever seeing the light of day before they’re obsoleted or pushed off indefinitely. For collaboration, we maintain a separate ticket system with to-do items (class #10: things other people have to do), but those tickets are usually higher-level while the tasks that go into OF are much more detailed and very specific to me and how I work.

The single most important aspect of any task management system is priority management. I need to figure out what I’m doing next, and ignore everything else until I’m done with that and have to figure out what to do next next. If your tasks are clear-cut enough that they will naturally form a priority queue, OF can probably help you manage that and tell you what to work on next. I often have to re-evaluate anyway, so that’s not as helpful for me. I use OF to keep all of my work-related tasks. OF especially has a LOT more power to slice and dice your task list than I even come close to using, but I feel like I have a pretty good handle on my task list even with just the features I’m using.

While it lacks a good organization scheme for larger-scale projects, I’m finding that Apple’s Reminders.app is very handy for keeping non-project-related personal tasks. It feels like a to-do list, but it also has reminder times (limited but helpful) and location-based alerts (which sort of work, but generally aren’t granular or flexible enough to be useful for me). It has the critical ability that I can easily share lists with my wife, and if you have an iOS device or Mac, it’s already included. For many people, that may be all you need, but I find the lack of hierarchies limiting in my organization. I use it for things like remembering to return items to stores, make appointments, and print photos. I use Reminders for some personal items in classes #2-#4, as many items as I can shoehorn in from class #6, and a lot of class #7.

A to-do list is a simple ordered list of items.

Clear is by far my favorite example of this. It lets you keep multiple lists, reorder them easily, and it has a very pleasing UI and sound effects. I use it primarily for keeping items in class #8. This will often encompass things I have to collect before leaving the house, or things to buy for a party. The flexible UI makes it very easy to adjust the list as I go, priorities can be adjusted by moving items up or down, and having it in a separate app allows me to concentrate on just that next list of items without worrying about anything else. Listbook is also a nice app for this, but the UI for Clear just blows it away. In the latest version, Evernote has some basic to-do list capability (which they call checklists), but I don’t find it useful for that much.

I consider a checklist to be a special kind of to-do list, where there’s a template for items, and every time you start a new iteration, you have to do everything (or most things) on the checklist.

Class #9 is the checklist. I haven’t yet found a checklist app I like. I would use this for managing the scope of repetitive actions, like test plans or deployment checklists. In the past, I’ve used OmniOutliner for this, but it’s not great (and has no iPhone version). I like OO for making outlines, but a checklist isn’t quite the same thing. I’m trying out TaskPaper (I picked it up recently on sale) and Workflowy (someone sent me a referral code) for this, but I haven’t had a chance to give either of them a whirl yet, though they seem like they won’t offer too much advantage over OO. They all still have the standard “make a list, save it as a document, then copy that document” workflow, which is substandard. I’d really like to find some checklist manager which can easily manage a bunch of different lists and “deploy” them as needed with the options to run a whole list at a time.

Certainly anything kept on a to-do list could fairly easily be kept in a plaintext file or a note app as well, but it’s nice to have a dedicated app with an intuitive UI that lets you easily reorder items, insert in the middle, and check items off individually as you go. These are the differences between a tool that can do what you need and a tool that does what you need.

I’m sure I have much more to say on the topic, but I hope that helps a little.

Kickstarter status check

Thu, 13 Dec 2012 16:44:10 -0500

To date, I’ve backed 15 kickstarter projects, all of which have been funded. Of those 8 have delivered. I’m not going to go into detail on all of them for various reasons (some are gifts that I don’t want to spoil), but here are details on some of them.

On time or as close to it that I didn’t really notice:

The Cosmonaut, a wide grip stylus for touch screens: Drawing on an iPad is necessarily low-res because of the large touch targets meant for fingers. There are some workarounds. This stylus embraces the large form factor and turns it into more of a whiteboard marker style. I like it. I don’t use a stylus very often, but this one works well for when I do. I don’t know if I’d buy it again, but I like it.
Coffee Joulies: They’re little blobs of stainless steel with some kind of wax in the middle that you put into your coffee, and they bring the temperature down to the drinkable range quickly and then retain heat to keep it warm longer. Marco Arment was famously unimpressed. I have found that they work really well for me. Using my regular Contigo cup, my coffee was previously way too hot to drink for about 20 minutes unless I added an ice cube, which had the side effect of making it get cool much faster. With the coffee joulies, it is drinkable in about a minute, and stays warm noticeably longer. I don’t know if I’d buy them at retail price - the backer discount was substantial. They are admittedly expensive, but that doesn’t take away from the fact that they work for me and I use them every day.
TiGr Titanium Bike Lock: I love this thing, though it has a few hiccups. There’s no place to keep the lock cylinder when it’s not used. This model is not quite long enough to lock up both wheels and a post, because my hybrid bike has a long wheel base and slightly wider wheels than many bikes (John has graciously agreed to let me exchange it for a longer one when available). It is however, really flexible, and has enabled me to lock my bike up much more easily in some places where it wasn’t previously possible. They can be easily chained to get around large objects. It is also very light. Expensive, but not compared to my bike. I would definitely buy it again.
Dark Sky for iOS: A high-precision weather prediction app that will tell you with good accuracy whether it’s going to rain in the next hour, and about for how long. I use this ALL THE TIME. Buy it.
Find my car smarter: A bluetooth 4.0 low-power dongle with an app that pings in the background to automatically keep track of where you left your car. Useful, and it seems to work, but we don’t drive much. I don’t regret backing this, but I probably wouldn’t buy it again.
Smarter Stand for iPad: I absolutely love this thing. It’s a set of clips for the iPad Smart Cover that open up new configurations for how it can stand up. Great idea, great implementation. Buy it.

And some delivered but not so on time:

Elevation Dock for iPhone: I like the Elevation Dock, though it’s been plagued with problems. It was released (late) just before the iPhone 5 changed the connector, which annoyed a lot of people, and the solution thus far has been a hack. I still have my iPhone 4S, so that’s not an issue for me yet. But I’ve found the circuit board to be flaky, and have had to have both of them replaced. Their support email has a turnaround time of days, which is unacceptable. I still like this, and I use it, but I would not buy it again.

And some that I’ve not yet received:

Pebble: E-Paper Watch for iPhone and Android: I’m not really sure why I backed this. It seems like a great idea, and I have no idea if they’ll pull it off. I hope it’s great, but it may not be. It’s late, but the founders clearly opened up a vein of latent interest they weren’t prepared to deal with, and they’ve been sending reasonably frequent updates of progress. They asked for $100,000 and raised over $10,000,000. I can forgive this for being a totally different project than they thought they were getting into.
The Nifty MiniDrive: Also late, but it seems like they’re actually very close to shipping. (Update: Received a shipping notification the day after I wrote this!)
Double Fine Adventure: It’s a Tim Shafer adventure game. No-brainer.
Shadowrun Returns: Shadowrun was one of the best RPG systems I ever played, and I love the universe. Clearly I was not alone. Ambitious, but they raised a lot more than they asked for, and have been delivering steady updates.

Honorable mention:

I missed the kickstarter campaign for this by a day, but I bought one from them directly. It’s a really well-designed leather bike handle that makes carrying your bike up stairs much easier. Walnut Studiolo Bike Frame Handle.

So far, no real doozies, and few things that I use and enjoy often. I’m pretty happy.

Wifi performance and the new iPad (counterintuitive Airport settings for advanced users)

Sun, 25 Nov 2012 20:24:02 -0500

The iPad 4 (and iPad mini, though I didn’t test with one) claims to have up to twice the wifi performance as previous models. While trying to configure my Airport Extreme to get the best performance, I discovered some interesting and counterintuitive results about the settings. (This probably also applies to the newer Airport Express, though I don’t have one.)

tl;dr: The Wireless Options screen is a performance deathtrap. Leave everything here at the defaults for best results. Most changes on this screen will degrade performance.

(UPDATE: Added iperf results to the bottom.)

The longer answer: it seems that you get the absolute best wifi performance (for iOS devices, this is 150Mb/s, but laptops can go faster) if you let the router determine the protocol (radio mode) and channels to use automatically, and set the 5GHz network name to be the same as the 2.4GHz network name. I’m measuring performance by the speed at which the Airport is telling me my devices are connecting (which seems to correspond to real-world throughput).

Update: These tests are relatively informal, but I’ve been able to repeat them several times. I’m definitely curious if others are seeing the same results. I have tried running the Ookla speedtest app to verify, but at the moment, I don’t really trust its results. In theory, at a 150Mbps data rate connection, I should get 50Mbps down (the speed of my cable modem), and I’ve never seen it top 20 on the iPad. I do get 50Mbps on my wired desktop. I’ve seen speedtest results vary widely by which server you’re attached to, and I can’t figure out if there’s a way to change that on the iOS app, so that may be the difference. It does seem to vary with the connection data rate speed, but not enough that I’m comfortable with it.

First, the network name. If you specify a different name for the 5GHz network, they will appear as two separate networks in the settings of the iPad, forcing you to choose one or the other to connect to. In this case, even choosing the 5GHz network explicitly gave much worse performance. I gather that the protocol enhancements they’ve done require channel bonding across both the 2.4GHz and the 5GHz range, and if you choose one or the other, you give up that option. Even signing in to both networks did not reliably improve performance. Switching back and forth did momentarily increase throughput, but not to the highest levels.

Here’s what happens to the iPad wifi data rate if you have a separate network name for 5GHz.

Connected to the 2.4GHz network, it’s not terrible, but we should be able to do better:

If you connect to the 5GHz network, it’s surprisingly even worse (it started out at 150, and rapidly deteriorated):

Second, the radio mode.

The default radio mode options screen looks like this:

But you can get additional options by holding down the (wait for it) Option key while clicking the radio mode:

I had previously had the radio mode set to all 802.11n, because I have a separate network for the few remaining 802.11g devices I have, and I thought I was benefitting myself by preventing the network from dropping down to g speeds to accommodate those devices. This turned out to be a counterintuitive mistake - if you lock the radio mode to 802.11n, it CANNOT achieve maximum throughput, and it cripples most modern connections. The only acceptable radio mode I’ve found is 802.11a/n - 802.11b/g/n (Automatic). After making these changes, nothing is preventing slower devices from connecting to your network. If you want the maximum throughput, you’ll need to manually ensure that everything can use the faster connections.

With the router set to n-only, here it is connected to the 2.4GHz network:

And the 5GHz network again tanks very quickly (and having no separate 5GHz network name here seems to make little difference):

I’ve only tried it a few times, but it seems like in any configuration other than fully automatic, connecting explicitly to the 5GHz network may appear to give fast performance, but it will quickly step down to something slower. The only way I’ve been able to get a sustained 150Mb/s connection is to leave the Airport Extreme in fully automatic mode.

Here’s what it should look like:

As a bonus, here’s a 2010 Macbook Pro hooked up to the same network in fully automatic mode:

Update: and a Retina Macbook Pro:

Turning the router back to n-only mode also craters the rMBP data rate:

At the suggestion in the comments, I’ve tested this with iperf, and it has completely substantiated my results. Using fully automatic, I get a reported actual 70Mbps transfer rate from iperf, and that’s the only setting that remains high over time.

Here I started out on automatic (60-70Mbps), then dropped down to N-only (barely 1Mbps), then split the networks and connected to the 2.4GHz network (25 Mbps):

Choosing the 5GHZ network gave good performance initially, but then it tanked to below 1Mbps, and didn’t go back up.

Repeated testing with the fully automatic network over time showed a sustained high rate:

Resist the urge to tinker with these settings.

iCloud Documents and OS X 10.9 wishes

Mon, 19 Nov 2012 10:16:26 -0500

OS X 10.9 is coming. Here’s are some improvements I’d like to see:

* First, fix iCloud Documents and autosaving. This is by far my biggest gripe with the current ecosystem. There are too many gotchas, and it’s too confusing. Here are the problems I see:

If a document lives in iCloud, the only way to access that document is through the application that created it. I understand that for security and usability reasons, it makes sense to only allow applications to access their own document collections, but this restriction should not apply to the Finder. There should be an iCloud documents section in the Finder, where you can see everything, move documents from one application to another, drag and drop them normally, and share them with other programs and people. This section should also be a local cache, so you have your own copy of the documents if the network is unavailable for some reason.
I love autosave, but it’s too confusing. Save As… should not be an afterthought, it should be restored to a default menu item, and it should _always_ keep the original document as it was the last time it was explicitly saved or closed. The “Duplicate” menu item is, as far as I can tell, useless and should be removed.

* Make shared photostreams writable by multiple people. If both my wife and I could easily publish a shared photostream of pictures of our kids, it would make all of the relevant grandparents extremely happy.

* Fix Mail.app for good. It still needs to be restarted periodically because it forgets that there are servers out there somewhere. I’d also like to see it be easier to use multiple email addresses on the same account.

* Add customizable search engines to the Safari search options.

* Restore the ability to display RSS feeds in Safari. I still haven’t seen an rationale for why this was removed.

* Launchpad is dumb. No one uses it. (Update: I’ve been informed that some people do use it. I maintain that it adds little to the experience for anyone who doesn’t already know how to get around.) Get rid of it.

* Make fullscreen mode for apps work with multiple monitors.

* The obvious step of bringing Siri, iBooks, and Maps to the Mac.

* Something better for menubar icons. They’re useful, but difficult to manage and there are an increasingly large number of them.

[ Update: I thought of two more:

* Notification Center should be updated to show the full archive history of notifications, and allow the option of moving notifications to Reminders.app.

* Periodic reminders as an option for Reminders.app. Something like “Remind me again in X minutes/hours/days/weeks if I dismiss a reminder.

]

You can use the feedback form to submit ideas.

Yet another hiccup on the transition to the cloud (iTunes restore is half-hearted)

Fri, 16 Nov 2012 10:28:56 -0500

Since the beginning, it’s been the case that when you restore a backup of an iOS device from iTunes, it doesn’t restore the apps as part of the backup. That makes sense in some ways - the app data is completely separate from your personal data, and the apps can always be restored from the store. In the past, this has always worked fine for me, and in the past few revisions of the OS, it’s even flawlessly maintained the position of all of the apps I had installed. With the advent of encrypted backups, it’s even stored saved passwords with the backups. This is good - the idea of restoring from a backup is to get you back to exactly where you left off in as little time as possible.

But this time around, there’s a new wrinkle. Without fully realizing the implications of that choice, on my last iPad, I had turned off the syncing of apps with iTunes in favor of managing the apps directly on the device. That seemed like a rational choice at the time - there was little reason to manage my apps in iTunes when I could just as easily do it on the iPad. However, as it turns out, if you have app syncing off, and then restore from an iTunes backup, what happens is this: you get a freshly restored iPad with all of your data, and exactly zero apps installed. Apparently, the ONLY way to get them back is to painstakingly go through your purchased list, figure out what you actually had installed, and bring them back one by one. And if you happen to forget something, you can end up with a lot of orphaned data in your Other section where the app’s data still exists on your iPad but the app doesn’t (the only way to delete that data is to magically figure out which app it might be, reinstall it, and then delete it again). Moreover, once you turn off sync apps, there’s no evident way to turn it back on again without completely wiping the list of apps that are already on your device and starting fresh. With a few hundred apps, this is a world of hate.

Incidentally, restoring from an iCloud backup does do the post-restore app sync, but with two important differences: 1) it throws a large number of “restore incomplete: some items could not be restored” errors with no information about what actually failed, which is not very confidence inspiring and 2) it does not store most passwords. This last is a nice security issue if someone happens to get access to your backups, I guess, but it is hardly what I want out of restoring a backup. What I want is to get back to where I was without a lot of effort. Previously, restoring from an encrypted iTunes backup was the best way to do this, but it’s now hopelessly broken if you take advantage of the iCloud features of starting to separate from iTunes. (Update: A number of people have asked what I’d like to see here. The ideal solution for me would be to have the saved passwords stored with the iCloud backups, but encrypted separately with a different password/pass phrase, with no recovery option for that separate password. If you lose the password password, then you can still recover your backups, but your saved passwords are gone and have to be re-entered. That’s a hassle of course, but it leaves you in no worse a situation than is the default now with iCloud backups.)

Apple is trying to push people towards the cloud, which is fine when it works, but this is a very big functional hole they’ve left open. I’m disappointed in the user experience here. iCloud suffers from a large number of these kinds of transitionary elements where it’s not sure what it wants to be or how to get there. It’s time to rip off the bandaid.

Initial impressions: the Kindle Paperwhite falls into the Uncanny Valley

Wed, 03 Oct 2012 16:08:39 -0400

I really want to like the Kindle Paperwhite. It’s a beautiful piece of technology. I’m going to give it some more time before I decide, but my first impressions are almost overwhelmingly negative compared to the previous model I have (the $79 “plain” Kindle), even about the changes that have some benefits.

Some specifics:

When reading longform (books), I find that I quickly get incredibly disoriented about where I am in the book. Every visual cue here helps, and modern e-readers offer very very few of them. If I don’t have enough that are persistently on the screen, my brain gets preoccupied with trying to figure it out, and it’s very distracting. Indicator cues that I have to tap some area of the screen to make appear are not an adequate substitute - in order for them to work for me, they have to be on the screen at all times. The old Kindle had a visual progress bar and a percentage meter. The new one drops the progress bar in favor of some options of “time left to read the book” (useless), “time left to read the chapter” (useless and largely inaccurate), or “location number” (useless and obscure). There is still a percentage meter, but that alone isn’t enough. Similarly, there’s no way to persistently display the (pleasantly small) clock on the screen, but that’s not a new problem. Incidentally, iPad reading apps pretty much all universally get this wrong. Recent versions of iBooks are slightly better. Recent versions of the Kindle app are worse. I get that some people want the chrome to fall away so they can “concentrate on the text”. I feel lost when that happens, and I’m not alone. Forcing me into that mode makes me not want to use your app at all. This is my #1 dealbreaker.
I miss the physical buttons. The touchscreen is reasonably okay, but it’s still too easy to accidentally turn pages. The textured screen is nice.
The front-lit screen is, in fact, lovely to look at. But in being so, it’s almost-but-not-quite too good, and it draws into sharp contrast every irregularity on the surface of the screen. This is the Uncanny Valley of display technology - it’s not perfect, and in being so every flaw stands out. It’s possible that I have a defective screen (a replacement is on the way), but while most of the screen is evenly lit, there are subtle dark/light blotches near the bottom. They immediately stand out. In a completely dark room, the front-lit display is actually harder to read than the previous Kindle with a light. Yes, the illumination is diffused and evenly distributed over the screen, but that results in the page being a glowing rectangle in the dark with no context. Removing the light spillover on the outside case increases the contrast with the surroundings (since the screen isn’t large enough to fill the entire field of vision), so rather than disappearing, all you notice is the edges of the screen. I find this very distracting. In medium or bright light, the front-light actually really helps increase the contrast, but then the large reflective “kindle” label on the front draws your eye. Someone didn’t think this through properly. I find the black border to be more distracting than the old grey one, but that may just be that I’m not used to the difference yet. In many ways, the screen is better than before, but I haven’t found it to disappear the way the old one did. In terms of eyestrain, I think it’s about equivalent to using the old one with a light, except you don’t need an external light. I definitely want to keep the light on all the time, as it makes the screen much higher contrast and easier to read. I haven’t given it an extended test yet.
I really like that they added the ability to have a lock on the store purchases without locking the device, but mysteriously even if you have no lock code set, you still have to “swipe to unlock” AFTER pressing the power button. If you have a lock set, you have to press the power button, enter the code, and THEN swipe. This is annoyingly tedious without any apparent explanation. I also don’t really understand why the store lock is under parental controls, but that’s a minor quibble of terminology.
The light, while pleasant in the daytime, is entirely too blue for nighttime reading. I think it’s messing with my sleep in ways the old one doesn’t. Someone at Amazon needs to read (or reread): http://stereopsis.com/flux/research.html
The black matte frame is a fingerprint magnet.

I think that’s about it. I want to like it, but I think I don’t. The more I use it, the more I want to switch back to the old one. That’s not a great sign. Maybe it’s just me.

(Update: After using it for a few weeks, all of these initial impressions held, and I returned it in favor of keeping my old Kindle.)

Maps. It's not just about the data.

Fri, 21 Sep 2012 12:42:00 -0400

As you may have heard, people aren’t happy with the Maps app in iOS6. I think the problems I have with it are a little different from the standard map glitches.

There are data problems, to be sure, but I expect those to be fixed reasonably quickly, as people start to use the app and Apple can collect problems and fix them. (BTW, instead of just complaining, report data problems. Every pin on the map has a “Report Issue” button.)

Most of my problems are a little deeper. The old maps app was a showcase - when I wanted to show people what the Retina iPad was really capable of, I pulled out the maps. It could display tons of data on the screen with razor sharp text (even almost too small to read but still legible), and scrolling and zooming was nearly instantaneous. On wifi or LTE, it never took more than a second or two to fill in all of the tiles on the screen in hybrid mode.

That snappiness is simply gone. The overall performance of the app has dropped precipitously, and it now often takes 5-6 seconds for the tiles to draw after each move. It looks like there’s significantly less local caching going on, and I see a lot more of the holodeck placeholder background. I get that they’re doing more, but Apple has historically been on the side of doing less better, and I think a poor tradeoff has been made here. (Update: I think the performance issues only apply to using the hybrid view. Even on the iPhone 4S, the standard view is fairly snappy. But I use hybrid view the most, and poor responsiveness kills the illusion that you’re handling a real object.)

On top of that, the app shows a lot less data on the screen at each zoom level. Before, there were a great deal of landmarks pictured at even moderate zoom, enabling _reading the map_. Now, if you don’t know a much more precise location of what you’re looking for, you’re lost and relegated to search. Here’s before and after. Notice how few streets and avenues are labeled in the new version (bottom), and how Central Park has no internal landmarks other than the obvious large bodies of water. This is largely useless to someone who doesn’t already know how to get around.

(Click through to see the full resolution image)

I want my Retina apps to take advantage of the display to show me not just beautiful pictures and crisp text, but LOTS of it. I want information density. I can’t see the argument that the old maps app had too much information displayed on the screen at once, and if you can, then what I want is a slider to show more or fewer labels.

There are other more minor usability oversights, like the fact that it’s impossible to switch from driving to walking directions without completely backing out and starting your directions search again, and there’s no evident easy way to re-orient the map to north-is-up once you spin it. That stuff I can live with, but I’m very sad that they’ve taken a glorious impressive showcase map and turned it into something lesser.

Let me be clear - I think we owe a big thanks to Apple’s map team for the hard work in making this effort - better competition and less being beholden to any one provider of this data is always a good thing in the long run, and I believe them when they say it’ll get better. But at the moment, the product isn’t there yet, and it’s disappointing that we needed to take a few steps back to get here. This is a product that people use and depend on with great regularity, and having it work three months from now doesn’t help if it won’t do what you need now. It certainly hasn’t held me off from upgrading to iOS 6 (which seems like a massive improvement in every other way), and it’s not a good reason to flee the entirety of iOS. I just hope this formerly outstanding product which is now merely very good gets restored to its former shine, and soon. I mean - let’s put this in perspective - it’s an incredibly detailed map of the entire world that you get to carry around in your pocket basically for free with your phone that you were going to buy anyway. That’s pretty special.

Some thoughts about App.net and social networks

Fri, 10 Aug 2012 23:40:00 -0400

First: Twitter. I like Twitter. I use it daily, and it’s become invaluable to me for three reasons: 1) It’s a great realtime news network, both for discovery and searching, 2) I’ve had some interesting and fun conversations, in short bursts, and 3) For some reason, companies, especially small ones, respond much faster to general support requests issued via twitter than almost any other medium. It took me a very long time to see the value of Twitter, and I had to cultivate a network of people to follow who would deliver me the news I actually wanted to see and the interesting thoughts I wanted to respond to. I suspect that most people won’t do this work, and those are the people who don’t see the value of Twitter. Almost none of my actual friends are on Twitter. Most of them still accept the “tweet what you’re eating for breakfast” canard.

Second: Facebook. My actual friends are on Facebook. My relatives are on Facebook. Everyone is on Facebook. Facebook’s great triumph is that somehow, millions of normal humans who are otherwise completely incapable of handling whatever passes for a “REGISTER NOW FOR A FREE ACCOUNT” form on the internet have managed to acquire Facebook logins and use them on a regular basis. I do not ever expect this to happen again. Facebook is a part of the landscape, and I don’t see how it can be challenged at any time in the near future. But it doesn’t have to be. Other services can happily coexist with Facebook, and those who want them will use them.

Facebook is known to be mining personal information, they change stuff all the time, and they’re a fairly constant background in the social network space. For reasons outlined in the previous paragraph, they’re not going away anytime soon. Since they’re so out of reach, I don’t see that they’re in play here, and not really relevant when talking about a new social network. Twitter, on the other hand, seems vulnerable, despite the big push they’re getting from Apple via OS integration. As a user, I find it fairly easy to ignore most of the various advertising pushes, but in general, I tend to view advertising as a nice way for companies to waste their money on something I’m just going to ignore (I keep searching for a way to prove that this is actually true; no one seems to believe me when I say I’m almost completely immune to advertising). As a programmer, I’m fairly dismayed by the turns that Twitter has taken to upset their developer community. Their third-party clients are what makes the platform tolerable, and I hope they come around to seeing that. If I had to use Twitter on the web, I possibly wouldn’t. However, I think even that is largely irrelevant. Twitter is vulnerable not by nature of alienating current users, but by what they don’t want to be.

When I first started using Twitter, I thought it would be hard to express fully-formed thoughts in 140 characters. As it turns out, we’ve gotten much better at that and some of the underlying metadata changes have helped, but it’s still fairly limiting. On the other hand, I immediately thought that 140 bytes would make a great size for a large distributed message passing mechanism for programs to talk to one another; messages not meant to be seen by people. For all I know, people are actually using private accounts for this, but I envisioned that these would be public messages, publicly addressable, but by your machine to my machine instead of you to me. Maybe it would be chess moves, or picture sharing, or whatever, but your app could find and direct messages to me just by using the already public mechanism instead of forcing me to register for YET ANOTHER network. These messages might involve text, and they might even involve character limits, but they’ll carry a lot of other payload that’s not meant to be human-readable, and will be interpreted by the endpoints. The real play here is that you’ll be using an application that isn’t “app.net”, but it will use that account to find you and communicate with you. Conversations will happen wherever there are people, but it’s still very hard for machines to find each other properly.

Enter app.net. A lot of people seem to think that the whole approach is either folly or outright disingenuous (“bash twitter, and oh, by the way, we’re starting a competitor”), but it seems to me that that’s not what’s going on here. I’m not privy to the backstory other than what I’ve read, but it looks to me like he was building some other product, got fed up with platform issues, and decided to take a gamble and pivot to a pure API play if there’s sufficient interest in the form of a kickstarter-esque backing campaign. If the project doesn’t get “funded”, I suppose we’ll see what happens. It will have been an interesting PR exercise in any event.

The API infrastructure messaging plan is big, and it’ll require a lot of thoughtful planning and work to make it actually useful, but that’s what I see when Dalton says he’s launching “a financially sustainable realtime feed API & service”. App.net doesn’t have to kill Twitter to be successful, it just has to move into the space that Twitter never really found itself at home in. I hope he manages to pull it off. That’s why I backed it, and I think you should too.

https://join.app.net/

(I’m @fields on app.net and twitter. Don’t bother finding me on Facebook if you don’t already know me.)

Mountain Lion seems to have (partially?) addressed the memory management issues in OS X

Thu, 02 Aug 2012 10:14:00 -0400

Having installed Mountain Lion on several machines, turned the dynamic pager back on and run with roughly the same pattern of applications, I’m now finding that performance is vastly improved. I have not been able to replicate the same sorts of drastic slowdowns I was seeing under Lion, and I barely notice when Time Machine kicks in. However, I’ve heard reports that the VM subsystem still falls over if you push it too hard (perhaps on machines with less physical RAM), and when it does, it exhibits the same sort of cascade slowdowns that were present in Lion. It’s possible that they did not fully address the underlying issue but did tune the settings so it happens less often if you have more RAM. This is a significant improvement - under Lion, adding more RAM did not help - but if you have “insufficient” RAM, that’s likely to be not much comfort. Since I installed ML, I’ve seen zero pageouts, which is as it should be - there’s no reason for the OS to swap out if it’s got enough physical RAM.

At this point, it’s hard to make a conclusive recommendation - I’d suggest trying it both ways.

My impression is that performance in Mountain Lion is improved across the board, but that’s what I thought about Lion when it was first released. We’ll see if it holds.

The Uber Ice Cream Promotion is deliciously brilliant, and keeps getting better.

Sun, 15 Jul 2012 11:17:51 -0400

The Uber guys are PR masterminds. I am in awe.

Step 1: Take a service that’s beleaguered on all sides by impending legislation.

Step 2: “Pivot” to a new service - premium ice cream delivery trucks on demand! - that people are sure to love. Bravo! It’s a sure hit, but it’s so outrageous that no one really expects it work.

Step 3: Launch with insufficient capacity, but that’s ok - how could you have possibly predicted the demand for ice cream on hot July Friday?

Step 4: Apologize and offer a $15 off coupon towards your “regular” service as a consolation.

Step 5: Sway public opinion towards you (reminder: Uber == ice cream!) PLUS get otherwise reluctant new customers to try out your service.

This is so far superior to just offering a coupon. I can’t wait to see what’s next.

Speed up the Dock animation in Mac OS

Wed, 25 Apr 2012 17:12:10 -0400

ben:

I don’t really use that at many defaults write terminal commands, but I read about this one recently and I like it a lot.

If you hide your dock, you can speed up the hide/show animation by using the following command in Terminal:

defaults write com.apple.dock autohide-time-modifier -float 0.5;killall Dock

This will make the animation much faster. You can also get rid of it completely, but I think that looks weird.

Some followup on disabling the dynamic pager in OS X

Tue, 24 Apr 2012 16:16:04 -0400

Some additional uncategorized thoughts on disabling the dynamic pager:

This is not a magic bullet. Anything that actually needs to use the disk is still going to take place at disk speeds. But it should take a whole lot of load off of the disk.
It’s evident that not everyone is seeing the problems I’ve been seeing, but there are enough people who have to make me think there’s a real issue here. I can’t depend on my own findings being representative, because I push my machines hard - I want to have everything open at once, converting, transcoding, browsing, whatever. I’ve been known to keep over 200 browser tabs open at a time. In five different browsers. I have over 20TB of data on my main machine including backups, so…. thanks for all of the suggestions to switch to SSD (I have one, but only for applications and commonly used documents and cache - my homedir won’t even fit on it). This is why it’s good to know if other people are seeing the same problems, and if the same fixes alleviate them.
Some people have noted that the problem can be improved by adding more RAM. I found that it got worse when I bumped my Mac Pro from 24GB to 32GB, but that may just be coincidence. I also feel like it’s gotten worse in successive point releases of Lion (my first impressions of Lion for the first few weeks were that it had a performance gain over SL).
I love my Macs. I point this stuff out so that they’ll improve, not so you can try to convince me that your Toshiba laptop is better (won’t work; I’ve used them). My understanding is that their development teams are tiny, and they have to be understandably aggressive about prioritizing what features they work on. As a developer, and even moreso as a project manager, I have a lot of tolerance for that.

More testing is certainly needed. But given the overwhelming number of responses that are of the form “yes! this really helped and made my machine much faster and more responsive!”, I believe that the responsibility for doing that testing does lie with Apple.

One particularly weird thing - I seem to still be getting page outs over time (or something else is reporting as a page out, though I have no idea what that would be). Here’s what my Activity Monitor looks like:

As you can see, there’s a lot of wired memory (though it does get freed; I don’t have the problem that some have reported where wired memory just builds up. I suspect that means they don’t actually have enough RAM for what they’re trying to do). But there’s also a great deal more free and less inactive than before. But I’m still baffled by the page outs.

Something is deeply broken in OS X memory management (Lion performance problems part 3)

Wed, 04 Apr 2012 00:00:00 -0400

(Mountain Lion update: possibly fixed.)

I’ve been doing some investigation with Perry Metzger into what appears to be a huge problem with memory management in Lion (I, but not everybody, have seen this going back to Snow Leopard, though it’s worse in Lion. This may have something to do with the fact that I was running the 64-bit kernel on SL and most other people weren’t.)

The main symptoms that we’ve encountered:

Frequent beachballs, particularly when switching applications and sometimes even tabs.
General overall slowness and poor UI responsivness.
Specific and drastic slowdowns on every Time Machine run.
High memory utilization in Safari Web Content, mds, and kernel_task processes.
Large numbers of page outs even with a good deal of available RAM.
High amounts of RAM marked as inactive which is not readily freed back to other applications, with page outs favored.

The core issue seems to be that the virtual memory manager is bad at managing which pages should be freed from the inactive state and which ones should be paged out to disk (and, consequently, back from disk). There seem to be at least two distinct problems here, though it’s difficult to tell for sure without proper instrumentation - 1) program data is not well prioritized to remain in physical RAM over buffer data, and 2) the garbage collection algorithm may require that all of a program’s data be in physical RAM before collection can happen, causing extensive paging.

High disk activity from time machine and spotlight (mds) seems to cause a lot of pages of program memory to get thrashed out to disk to make room for the disk buffer pages. This manifests itself as basically every process waiting for the disk, an incredible performance drop across the board, frequent beach balls, very high amounts of inactive memory which doesn’t seem to get freed, and high numbers of page outs even with memory available. I was able to get some temporary benefit out of running the purge command to blow away the disk cache (see part 1 and part 2), but the problem returned after not much time. I’ve seen several blog posts recommending disabling the dynamic pager in Snow Leopard, but thought that was a fairly drastic step until Perry recommended it.

The results here are dramatic - after rebooting, both of my machines (mac pro with 32GB and macbook pro with 8GB) are running viscerally and noticeably faster. Many operations which used to have a delay became instantaneous - task switching, viewing thumbnails in Lightroom, switching windows/tabs in Safari. I didn’t realize until it went away just how much I’d been waiting for the most mundane things. Even under very high load, the entire system is very responsive now.

The only downside I can see is that things would likely start crashing if physical RAM actually runs out, but I’ve noticed that the footprint of all of my apps seems to be much lower. Just keep an eye on RAM (it helps to run Activity Monitor and make the dock icon show free memory - you can configure that by right clicking on the dock icon).

Ostensibly, this is “risky”, but so far the effects only seem beneficial. Likely the worst thing that will happen is that the OS will kernel panic if it completely runs out of memory, though it may not even be that bad - there seems to be some speculation that the system may turn on the dynamic pager automatically if system memory gets too low; I haven’t encountered that, though I have noticed that if I top out physical RAM, the page out count does go up. I have no immediate explanation for that.

I can’t recommend this as a long term solution - Apple should fix this - but in the meantime, it seems to be having great results for me. Here’s how much of a difference it is: I was going to turn it back on to collect some comparative numbers for this post, but I couldn’t bear to cripple my machine again.

To deactivate the dynamic pager:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

followed by a reboot. You can turn it on again with

sudo launchctl load -wF /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

Some references: