Sean McGrath wrote a great article on the struggle for Open Source to manage complexity. It's a great insight because much of programming is managing complexity. This requirement has fostered "high level" languages and continues to be powerful concept that pushes the state of the art.
Open Source does have one tool it uses to manage complexity and that is competition. For a very long time, Apache was essentially synonymous with "web server". Now we have Lighttpd, Nginx and Cherokee. All these projects have their reasons for existing, but I'm sure the fact they couldn't simply mold Apache to their needs played a role. Starting from scratch is so appealing because it removes the complexity.
It is arguable whether starting or forking a project is the most efficient way of removing complexity. At the very least though it is one tactic. As Sean mentioned, the Open Source development process will end up increasing complexity. This is not an invalidation of the process, but rather a simple trade off. The question in my mind is what other tactics can mature Open Source projects use to help manage complexity? Is it a tooling issue or an organizational problem? I'd wonder if things like Distributed Version Control Systems and Bug Trackers might help in terms of reducing/increasing complexity. Are there other tools on the horizon to help calm the storm of complexity or is the issue just a fact of Open Source software? Hopefully bringing up the issue can help begin turning some of the many eyes towards finding the solution this "bug" of complexity.