PCI-X vs. PCI Express

PCI-X is the overpowering dominant I/O peripheral standard in servers today. In other platforms such as client PCs, industrial computers and communications systems PCI Express could find vital entry points, primarily because PCI-X has not broadly penetrated these platforms. Will PCI Express be able to displace PCI-X in servers? What are the prospects for such a transition?

Given the economic state of our industry, it is more important than ever to make the right choices about technology adoption and migration. In the decision about whether or not to migrate to a new or enhanced standard, cost and performance benefits must be weighed against the magnitude of infrastructure change that is required.

Clean Forward Migration from PCI to PCI-X

The sustained long-term migration from PCI-32/33 to PCI-64/66 to PCI-X was due in large part to a clean path of forward and/or backward compatibility at each step of the standard migration. This allowed all aspects of the infrastructure to ramp progressively during this migration which has already spanned more than 10 years, doubling peak burst bandwidth at each step up to the current PCI-X 133 and beyond. There are yet several steps in the forward migration of PCI that have yet to be realized in production, taking this standard to 266, 533MHz and faster. This will provide a performance path that will be in production in mainstream servers beyond the end of this decade.

The chart above represents the design windows for each phase of the standard. Each standard initially appears on the chart when the specification is agreed to and development begins. The ?ramp-in? is fulfilled when production starts and the standard become a requirement for new leading edge servers. The ?ramp-out? is a soft approximation of the point at which the standard ceases to be useful from a performance perspective, though it may continue to have a longer life for ?legacy? purposes. The time in between (excluding ramp in and ramp out) is an approximation of the useful life of that standard from a design win perspective (marked as ?# yrs? on each line of the chart above). EISA is shown for reference only ? it is not part of the PCI migration.

Server platforms adopt and deploy these standards much faster than client or other types of platforms. The only reason that this remarkable pace can be sustained is because these standards preserve backward and forward compatibility from generation to generation. Without this attribute, it would be impossible to sustain such a fast rate of migration between them.
The transition to PCI-X is one that deserves some comment. In 2001, R&D ramped for PCI-X at three speed grades. At 66MHz PCI-X quickly became a replacement for PCI 64/66 because it was much more robust and offered better throughput. At that time PCI 64/66 began a mid-life transition to PCI-X. Also at that time the 100MHz and 133MHz speed grades were introduced (shown together on the PCI-X 133 line).

The solid migration of PCI in the server space shown above carries with it the momentum of a massive infrastructure and a linear progression of compatibility from generation to generation. It has the inertia of a freight train, and cannot be easily derailed. The magnitude of this PCI based infrastructure for servers can be illustrated in terms of the number of slots in the installed base at any point in time.

This exercise attempts to estimate the cumulative growth in the installed base of high performance peripheral card slots in servers. By interpreting and merging a variety of market research information and market share information from various credible sources, we have synthesized a normalized estimate of the number and type of server I/O slots that are in the installed base, and a projection about how this profile is likely to change in the near term future.

While other new high speed I/O standards will also be utilized in these systems, they will be used for chip-to-chip connections rather than for expansion or peripheral interface cards.

I/O Standards Wars

As PCI has been steadily moving forward, numerous I/O standards wars have been raging around it on all sides, highlighted by the current contest between HyperTransport, RapidIO and PCI Express. These debates are so heated that it is easy to forget about the previous generation of server I/O wars just a few years earlier. Beginning in the late 90?s, the industry struggled over a series of failed server I/O initiatives that have since littered the path alongside PCI. These were known as NGIO, Future I/O and finally InfiniBand. While InfiniBand still has hopes of finding traction as an external system interconnect at the data center level, nothing has been able to successfully threaten PCI since its initial volume deployment in the mid 90?s.

Though InfiniBand was once promoted as a replacement for PCI, this strategy fell flat. In the aftermath of InfiniBand, Intel abandoned most of its hardware and software design complexity, preserving its serial physical layer interface specification, re-used as the foundation for PCI Express.

Could PCI Express Undermine PCI-X in Servers?

While server platforms are our main focus for this analysis, we should recognize that PCI Express is an attractive option for market segments that have not invested heavily in the PCI-X roadmap (such as client PCs), or for use as a passive high speed serial backplane for industrial computing.

It is sometimes argued that servers should move to PCI Express early because of performance. But when laid up side by side with the performance levels offered by PCI-X, PCI Express does not necessarily stand out as an overpowering choice.

While the chart above does not attempt to show precise time-to-market relationships between these two standards, we can be confident that PCI-X 266 will be widely supported in the server market long before PCI Express gains infrastructure momentum. The question is at what point server makers will find reason to carry these two migrations simultaneously.

Enabling 10Gbit Ethernet - The Litmus Test

The chart above is based on the maximum theoretical bandwidth of the I/O standard in either direction. One additional and important point that can be understood from this chart is that PCI Express 4X cannot satisfy the peak theoretical bandwidth of a 10G Ethernet. 10GE requires 1.25 GB/s bandwidth in each direction. PCI Express 4X offers only 1GB/s in each direction, resulting in a 25% performance inadequacy for 10GE. In contrast, the bidirectional nature of PCI-X allows instantaneously adaptive bandwidth allocation in either direction. Thus PCI-X 266 delivers 2GB/s for upstream or downstream traffic, satisfying the peak performance levels required for 10GE.

Dynamics of Migration

In the past it has proven difficult for the market to try to sustain two simultaneous infrastructure migrations, particularly for something as complex and multi-faceted as I/O standards. This is something that the market does not try to do unless the ?new? standard has an overpowering performance, cost and/or time-to-market advantage.

Performance: The performance migration of I/O is necessary to enable the attachment of newer faster peripheral cards. The most prominent single example of this today is 10GE (briefly discussed above). For this analysis 10GE is the litmus test for performance. Since PCI Express 4X does not meet this requirement, the industry must bear the additional cost and time to market of using PCI Express 8X instead. This factor clouds the decision somewhat about PCI Express.

Cost: From the OEM perspective, there is a discrete cost burden to support two standards migrations simultaneously, rather than supporting just one standard. From the chip makers perspective, certain designers have reported that they anticipate a substantial die size premium in the shift from PCI-X to wide PCI Express configurations. It is also well known that there is not a substantial pin count difference between PCI-X and PCI Express 16X. In addition, PCI Express will carry cost premiums associated with manufacturing learning curves, R&D ROI recovery in its early stages, and carry higher support costs at the outset. It takes a couple years for these issues to normalize, which is also about enough time for the new standard to move to its ?second generation?. The ?second generation? usually offers better performance and tends to address whatever lingering technical issues may have existed in the first incarnation.

Time-To-Market: Time-to-Market is a multi-faceted issue. While R&D and validation might seem like the key to the critical path, it is really the development of the infrastructure that matters most. A new standard has not reached the market meaningfully until it has a critical mass of I/O slots in the installed base, combined with a competitive array of peripheral types and vendors to provide meaningful solutions. Plus, buyers of high end servers have standards of quality and reliability want want statistically proven reliability guarantees. The validation process for a new platform technology is an expensive and time consuming process for the OEM. Until a new standard has a history of production behind it, its reliability issues cannot be known with certainty. Interjecting such new technologies into the platform adds uncertainty to the reliability question. This leads to one of the common dynamics of the server market ? it is often early to adopt a new technology in R&D, but it cab be last to deploy that new technology in mass production.

Considering that performance benefits are not overpowering, infrastructure is a concern, costs will be high and that there is not a clear time to market advantage, taking the risk to support PCI Express as a secondary and unproven peripheral card standard in servers could turn to be a competitive disadvantage. We urge caution.

2nd Generation PCI Express: a Better Outlook

The exact specifications of the second generation of PCI Express have not been clearly decided at this time, but all indications point toward 6.25GHz signaling vs. the 2.5GHz signaling of the first generation standard. This will not only boost data rates significantly, but it also creates speed grade steps that better match the speed grades seen in other enterprise and communications I/O standards such as XAUI and SPI4. The new performance levels achieved by Gen2 also make for a more compelling performance transition from PCI-X.

Even though PCI-X 1066 is on the chalkboard, lets assume for a moment that PCI-X 533 may be the final generation of high speed parallel PCI in servers that achieves widespread adoption. From that point (perhaps 2004), a second generation PCI Express 16x with 6.25GHz signaling could arrive in time to provide a meaningful alternative to PCI-X 1066. We estimate that the 2nd Generation PCI Express could show up around the 2006-2007 timeframe, dovetailing nicely after PCI-X 533. If this analysis is correct, OEMs and server I/O vendors are safe to proceed down the near term PCI-X path, rather than assume the added risk and cost of supporting an secondary I/O standard that might not successfully take hold until later. For a peripheral vendor, entering the market too early can be more regretful than being a bit late to market.

PCI Express in Client PCs

For the sake of completeness, we wish to offer a few comments about how PCI Express fits in to the client space, including as a replacement for AGP. I/O standards migration in the client space has a different dynamic than in servers. For instance, even though PCI and PCI-X have a clean forward path, client PC platforms never adopted them because the performance was not needed and the cost was too high. Integration of high speed peripherals offered a better performance and cost solution compared to making changes to the peripheral bus, which was adequate for most non-essential functions. The only performance centric area that needed a fast I/O card interface was graphics ? which resulted in AGP. But of late, each doubling of AGP bandwidth (now up to 8x) has yielded less in terms of measurable performance ? perhaps just a few percent. Overwhelmingly, this proves that the bottleneck lies somewhere else.

Over time, PCI Express will be able to take client I/O performance to new levels. There is no critical path for this transition. It can and will happen however and whenever the market is willing. PCI as a legacy bus will not go away quickly, and that is what will allow PCI Express to slowly phase in without much commotion. AGP is another matter. It is unlikely that anyone will attempt to support both AGP and its PCI Express alternative in one platform. This makes for a hard, expensive and unpredictable transition, whenever it happens.

Summary

In these challenging times it must be a high priority to preserve and extend platform infrastructure in a compatible manner and to extract maximum ROI from each technology, product and strategy. It is an expensive and high risk strategy to prematurely attempt to replace an entrenched and successful industry standard. In this market, competition rewards bad investments and bad strategies very harshly and very quickly. System OEMs and peripheral makers must judge between the risk and cost premiums of a revolutionary strategy vs. the leverage and security of an evolutionary approach.

When looking at the comparison table below, you might be tempted to question whether PCI-X would be rendered superfluous. While PCI Express will mainly be around in desktop systems, PCI-X will remain the prevailing high-performance interface for high-end workstations and server systems. Finally, PCI-X 1066 will be able to provide up to 8.5 GB/s.