​We attended the CBRS Alliance event in Washington DC today, and by our rough estimate, about 350-400 people were in attendance representing groups such as regulators, legislators, lawyers, technology vendors, property owners, service providers, investors, media and analysts.  We were impressed with the widespread interest in the new shared spectrum technology and services running in the 3.5 GHz band that is now called “OnGo.”  We have researched CBRS for many years and found several acronyms and CBRS-specific terminology to be blossoming.  We found several themes at the CBRS Alliance event and a follow-on event at Federated Wireless, a SAS service provider, of special note: a) the OnGo experience will serve as a mold for regulators, operators and other interested parties not just in the US, but also the rest of the world, b) Tier 1 operators and WISPs appear focused on Fixed Wireless Access (FWA) deployments in CBRS spectrum, at least initially, c) many presenters focused on the “OnGo backhaul to gateways” use-case, at least as an initial opportunity, d) interested parties have a concern that PAL licenses may become very expensive when the auctions occur, and e) there were a very large number of devices supporting OnGo at this event.

Acronym soup.  The CBRS Alliance did its best to explain the various acronyms and how the various players work together.  It would take at least six pages to cover just the top-level details.  The idea here is that the 150 MHz of spectrum in the 3.5 GHz range was previously used exclusively by the US Department of Defense and is now going to be shared using a three-tier process, where the military (the incumbent) will have use of it when it needs, then private license holders will get next dibs (PAL), followed by general users (GAA).  Starting today, GAA users will begin use of the spectrum in the Initial Commercial Deployment (ICD) that was announced today, starting at 9 AM Eastern.  A group of service providers called Spectrum Access System (SAS) providers have been authorized to install radios on the US coastline that sense when the military is using the spectrum and send channel-use information to equipment that is operating in the CBRS spectrum.  These SAS providers will, therefore, coordinate the frequencies between incumbent, PAL, and GAA users.
​
Our view on why OnGo and “Shared spectrum” matters.  We expect that by sharing spectrum amongst various parties, more traffic can move across a smaller range of frequency than by using the more common method of auctioning off frequency bands to be used exclusively by one entity.  We estimate that shared frequency will carry ten-times more traffic than frequency bands licensed for the exclusive use of single entities.  Thus, it is for the greater good that this OnGo / CBRS experience go the distance and allow a public demonstration of whether multi-tiered shared spectrum can succeed or not.  Already, we have the experience of shared spectrum in the 2.4 GHz and 5 GHz bands used by WiFi – there is no doubt this has been successful; in fact, most public estimates show about 80% of smartphone traffic is carried by WiFi rather than cellular systems, all of which as of yesterday was carried on licensed spectrum.  At the CBRS Alliance event, guest speaker, US FCC Commissioner Michael P. O’Reilly said that based on the success of OnGo, he expects similar models could be applied to additional spectrum (and he implied this might the sequential order of launch): C-band (3.7-4.2 GHz), 3.45-3.55 GHz, 3.1-3.45 GHz and 7 GHz (which we understand is meant to be the same thing as what is being discussed at 6 GHz by the WiFi community).

FWA opportunity is front and center.  Charter and AT&T focused their comments on their plans to deploy fixed broadband systems.  AT&T shared some impressive statistics about the performance of recent trials using Massive MIMO cell sites using distributed RAN over CBRS spectrum, which is connected to indoor baseband over fiber optics to the radio sites and then connects wirelessly to customer premises equipment mounted at the roofline: it said it achieved 140x12 Mbps at slightly over one mile over line of sight using 20 Mhz channels. Charter discussed it had deployed its first commercial FWA in Davidson City, NC to rural locations.  It also discussed how it uses dual SIM systems to allow customer coverage to Verizon’s cellular network. Charter also discussed private LTE, neutral host, and Industrial IoT use cases.  The Wireless Internet Service Provider’s Association (WISPA) President spoke about its members’ enthusiasm for OnGo and explained that 100’s of WISPs used the 3.65 GHz spectrum and expects more will use the 3.5 GHz / CBRS spectrum.  Currently, WISPA says WISPS in the US have 6 million customers.

OnGo as a backhaul.  We detected a theme that seems durable: CBRS spectrum can be used by enterprises with far-flung operations to save costs by reducing the installation of wired / optical cables and associated infrastructure.  There was an impressive list of vendors who had equipment at the show, a number of which were gateway devices that made connections between CBRS and other well-known protocols such as Ethernet and WiFi, to name a couple.  While OnGo/CBRS support is not as widespread on devices today, IoT devices supporting other wired and wireless systems certainly are, the list of which includes WiFi, Zigbee, Bluetooth, Ethernet and more.  We were taken by how compelling some presenters made a case for using CBRS simply assuming a reduction in new cabling to enable new systems such as kiosks, surveillance, digital signage, farming, and so on.  Many of these examples would increase the deployment of existing protocols like WiFi, Zigbee, Bluetooth, and Ethernet, instead of reducing their demand.  The idea that OnGo/CBRS competes with existing systems may be incorrect.

PAL auctions.  Commissioner O’Reilly said PAL auctions are scheduled for June 25, 2020.  In our formal and informal interviews, we understand there is a growing concern that CBRS spectrum auctions could be aggressively pursued not only by existing Tier 1 mobile operators but also by other players, not least of which could include MSOs and maybe even “Big Tech” companies.  Since the 3.5 GHz spectrum is where many countries besides the US have begun deploying 5G services, making equipment in these frequency bands commonplace, there is ample reason to want to use this spectrum in the US.  Bidders may raise the price high enough that enterprises will choose not to compete, and won’t view the CBRS spectrum as attractive as they had hoped.  In this case, PAL would look quite a bit more like a typical licensed spectrum, similar to other auctions.

OnGo devices abound.  At the show, the following vendors had devices on show (see pictures):  Sercomm, MultiTech, Sierra WIreless, Zyxel, Encore, Cradlepoint, AMIT Wireless, Commscope / Ruckus, Accelleran, Bai Cells, Cambium, Samsung, Google, LG Electronics, Sequans, Telit, JMA Wireless, Motorola Solutions, Cisco, BEC Technologies, Ericsson, ip access, BLINQ, Comba Telecom, and Westell.

MACSec Helps Pave the Way to End-to-End Data Security

As consumers and businesses put more data in the Cloud, the importance of securing that data increases.  In just the last year, we have seen advanced threats and attacks by various entities to hack into that data and hyperscalers push back with both public and private mechanisms. Securing that data goes beyond just basic encryption or securing a server, and the role of the network is critical to better protection of data.

Many cloud customers are looking at providing end-to-end security to ensure, as best they can, that data can not be compromised.  MACSec plays an important role in the future on how networks talk to each other and how the secure transmission of data between different locations.  Security is especially important with 400 Gbps, as Cloud providers adopt 400 Gbps, it is not only being used for transmission within the data center but also Data Center Interconnect (DCI).  Cloud workloads will increasingly require secure connectivity between data centers.

Looking at the Ethernet Switch and Router markets, we project the percent of ports shipping with MACSec will increase significantly over our forecast horizon.  We expect vendors will continue to offer versions with and without MACSec, but as we move forward in time and have more purpose-built offerings for the hyperscalers that some products will only ship with MACSec.

The additional features and functionality included in Ethernet switches and Routers are positive for the industry.  It not only increases features which help grow ASPs and revenues, but it also increases the amount of Ethernet ports shipped by expanding the number of use cases.  400 Gbps DCI is a great example of feature and addressable market expansion.

By Alan Weckel, Founding Analyst, 650 Group.

There were two main announcements, a new relationship with Google Cloud Platform and a new flash device - the AFF A800. Also, in our interviews with NetApp, we learned about the future of Fibre Channel at the hyperscalers.

Google.  Google Cloud Platform now integrates NetApp Cloud Volumes as a drop-down menu capability as part of the Google console.  This allows enterprise customers, for instance, to use Cloud Volumes to manage their data on Google's cloud service while simultaneously managing their data on premise.  This relationship with Google now rounds out the NetApp relationships with the main hyperscalers - it already has in place relationships with both Amazon (AWS) and Microsoft (Azure).  NetApp Cloud Volumes on Google Cloud Platform is currently available as a "preview" capability (sign up at www.netapp.com/gcppreview) and is expected to go to commercial status by the end of 2018.  Customers will pay Google for the use of NetApp Cloud Volumes.

AFF A800.  New flash hardware available from NetApp, which besides having impressive density and low-latency capabilities supports NVMe-over-Fibre Channel.  Of course, the product also supports 100 Gbps Ethernet.  From a historical standpoint, it is interesting that NetApp, a company whose heritage was driven by storage over Ethernet, is touting Fibre Channel.  But, that's what its customers are asking for in order to accelerate their on-premise workloads such as database (Oracle), ERP (SAP) and other mission-critical enterprise workloads.  In our interviews with NetApp, we were told that Fibre Channel is growing faster than Ethernet - this makes sense given the company's foray in recent years to flash and low-latency workloads.

Fibre Channel at the hyperscalers?  We asked about what is going on with the hyperscalers' architecture to adapt to AI/Deep Learning workloads.  NetApp executives explained that AI workloads are different from traditional workloads; they are random, low latency workloads connecting to GPUs.  This type of workload, we were told by NetApp, works very well when attached to Fibre Channel.  From NetApp's perspective, if customers want to run AI workloads fastest, they would likely do so on-premise, using Fibre Channel.  Yet, many customers run their workloads on hyperscalers, all of which use Internet Protocol and the underlying Ethernet infrastructure.  We have always been skeptical that hyperscalers would adopt Fibre Channel.  We believe the hyperscalers may work with vendors such as NetApp to develop additional software capabilities to address the changing workloads relating to AI/ML/GPU workloads in the future - on top of IP/Ethernet infrastructures.  

The market is in a period of rapid adoption of higher speeds led by the hypserscalers.  The industry used 2016 and 2017 to adopt 25 Gbps and 100 Gbps port speeds based on 25 Gbps SERDES technology.  As we enter 2018, those same hyperscalers are about to adopt 50 Gbps, 200 Gbps, and 400 Gbps port speeds based on 50 Gbps SERDES at a record shattering pace.  In the data center alone, there are now eight unique port speeds, with countless more unique variations of form factor and pluggable distance.
 
The market will need additional bandwidth beyond what is currently available today.  Several of these technologies were highlighted at the OIF Forum conference.  100 Gbps SERDES will help drive the industry towards that goal.  Looking forward, 100 Gbps SERDES will help drive wave two of 400 Gbps, which will help enable Ethernet to extend its reach well outside of short reach data center distances.  At the same time, it will also have a long life, with use cases ranging from enterprise to service provider.
 
The big question often asked is why after so many years for the market to adopt 10 Gbps, will we suddenly see a more rapid pace of adoption going forward?
 
There are many reason why, but we should look at a few things are different this time.  First, the hyperscalers are a new type of customer.  Hyperscalers truly bring a new scale to networking and compute in a way that makes the traditional SPs look small.  Second, SDN, the hyperscalers have done something unique here that often gets overlooked that is occurring right now, in the second half of this decade.  Hyperscalers are increasing the utilization rate of their compute and networking resources.  For compute, this is approaching 100% utilization so the industry is in a period where hyperscalers, using SDN are able to grow network bandwidth at a pace faster than what the CPU is scaling.
 
This more rapid pace will not continue forever, but is one of the reasons why innovation over the next several years will occur more rapidly than historic norms and why it will be important for the industry to think about how to invest across speeds and technologies in order to better leverage existing investments.  If not, the pace of innovation will simply be too much to recoup investment in the compresses timelines we are currently in.

Broadcom joined both Innovium and Nephos by publicly announcing 12.8 Tbps fabrics with its Tomahawk 3 product line.  We love new data center silicon from all vendors, it is something we track closely and we see these as a disruptive technologies to the networking ecosystem and an enabler of next generation cloud architectures.  There will be many more such announcements in 2018.  Here are some of our takeaways as we enter 2018.
 
More rapid innovation cycle – Even noted in the Broadcom's Tomahawk 3 press release, we see the demand requirements of the hyperscalers as driving a more rapid cycle of silicon over the next couple generations.  Tomahawk 3 is being introduced less than the typical 24 months we see separating prior between generations of data center fabric semiconductors.  This will put significant pressure on parts of the supply chain, especially on optics vendors.  Optics vendors are still ramping for 100 Gbps and now must support both OSFP and DD-QSFP for 400 Gbps, essentially doubling their product diversity needs.  Not only are there more form factors, but there are also different variations of distance and specifications that increase the complexity.
 
What next – We see two waves of 400 Gbps, the first being based on 56 Gbps SERDES, the second coming in the 2020 timeframe based on 112 Gbps SERDES.  We believe 800 Gbps is not that far off in the horizon as hyperscalers like Amazon and Google continue to grow.  We note that the hyperscalers are about to be 3-4 generations ahead of the enterprise.  This type of lead and technology expertise really changes the conversation around Cloud.  We saw this at Amazon re:Invent with their Annapurna NIC, the Cloud is doing things that just aren’t possible in the enterprise, especially around AI, machine learning, and other new applications that take advantage of the hyperscalers size. 
 
2018, the Year of 200 Gbps and 400 Gbps – In 2018 we will see commercial shipments of both 200 Gbps and 400 Gbps switch ports.  We see significant vendor share changes because of this.  Simply put the Cloud, especially the hyperscalers will be that much bigger by the end of 2018 and they buy a different class of equipment then everyone else.  This will continue to cause the vendor landscape to evolve.

New CTO - It was great to see and hear Juniper’s new CTO, Bikash Koley at NXTWORK.  The message was very clear that new networks need to be built, not only from the speeds and feeds technology point of view, but from the operator point of view.  Scale and simplicity are very important and only going to be more critical as billions of IoT devices dump traffic and data on the network.  Juniper is looking to take the network it is building for some of the hyperscalers and Service providers and take help the tier 2/3 Cloud as well as the enterprise customer adopt a cloud architecture for the future.
 
Contrail – The big announcement was OpenContrail joining the Linux foundation.  What we saw in talking to customers and listening to the talks was that Contrail has significant commercial adoption, definitely larger than perceived in the marketplace.  We see the largest hyperscalers and service providers as having their own controllers, but see Juniper leveraging its expertise in this area into smaller Cloud and service providers.  This will give Juniper the opportunity to build, operate, and transfer for many customers. 
 
We were impressed with the caliber of customers that Juniper has.  Juniper has premier Cloud, Service Provider, and Enterprise customers.  We enjoyed listening to Twitter’s VP of engineering on stage as an example.  We see Juniper as one of just a handful of vendors that can support a customer base with this breath and complexity.  We note Juniper mentioned complexity and simplicity throughout the sessions; we only see networking getting more complex, especially as we move beyond 100 Gbps.  It is up to vendors to help the human scale with that complexity and we saw Juniper give many good examples during NXTWORK 2017. 
 

Today Nokia announced its new FP4 ASIC and 7750 SR Router.  Playing the leapfrogging game on speeds, we saw 36  400 Gbs ports in a 2RU box that looks awfully similar to a spine switch and the further blurring of what a next gen router and switch really look like, especially in the Cloud.  

We heard continued confusion over winning Cloud scale accounts.  We note that a customer like Apple buys from multiple vendors and for multiple reasons.  What Apple builds for their own consumption is not what they will deploy in a telco provider or peering location.  

The debate between merchant silicon and custom ASICs continues to come up.  While we are slightly in favor of merchant silicon, we note that the Cloud providers do not fear custom ASICs, they merely want to have standard APIs to control that equipment.

We note the Nokia ports are DDQSFP and not OSFP so we do not have a clear answer on form factor either.  We now wait for the next product announcement with the only clear answer that we are in a phase of rapid innovation in order to keep up with the network traffic demands of the Cloud.

Today Broadcom announced Trident 3.  The companies third major release of a chip that drove the merchant silicon revolution in the data center and started the white box movie in the Cloud.  With Trident 3, all of Broadcom’s data center switching ASICs now support speeds of at least 3.2 Tbps per chip.
 
Trident 3 is impressive, but a few things about it really caught my eye.  First, Trident 3 will offer five different skus, two of which are really focused on campus switching.  One could see a 48-port 2.5 Gbps switch out of the X3 version of Trident 3 next year.  We believe the Trident family moving into the campus will be significant for the industry once products begin to ship.
 
Second, native 25 Gbps ports.  Trident is the most popular of Broadcom’s ASICs, especially in the enterprise, and with Trident 3, we expect the market to quickly move away from 10 Gbps/40 Gbps products and towards 25 Gbps/100 Gbps products.  This product aligns well with our forecasts for this transition which we are excited to be publishing shortly.  We still don’t see a bandwidth need in most enterprises for 25 Gbps, but the ability to future proof at the customer level and the ability to consolidate skus at a vendor level will make this compelling.
 
Third, we see a potential for both switch vendors and customers to benefit from one family of ASICs from the campus all the way to the data center.  While it is too early to know the impact of this right after the announcement, we look forward to conducting interviews over the next few months to define this impact.

​It was great to catch up with old friends and make new friends at OCP this year.  The show was highly successful with attendance at the Facebook and Microsoft booths so large that it was difficult to move around.  On the switch side, most of the announcements were incremental to the market, but with new chips on the horizon, and a delay in 100 Gbps because of supply constraints, we see this as a temporary pause ahead of what will likely be some bigger announcements in 2018.

There were many highlights at OCP, but three things caught our eye while walking the show floor on both days.

•    Microsoft’s project Olympus server is about to transition Microsoft away from High-Density servers and towards Rack servers.  This is more in line with what other Tier 1 cloud providers are doing.  We note the smart-NIC is still a multichip solution, one that could be further reduced in future generations.  They also announced ARM based servers and joined Facebook on announcements in machine learning and AI optimized compute.  We see this change in Cloud architectures as a good sign for the industry.  The market is quickly moving into more use cases, which will help drive growth beyond just moving workloads away from the premise market. 

•    The white box vendors were in force at the show.  Edgecore showed various Fixed and Modular form factors.  We note that some of these boxes are modified for larger Cloud customers with the inclusion of large SSDs or memory.  We have a pretty good sense of what is using these additions, but that is a topic for a more detailed report.  We also saw Quanta and Delta with large presences on the show floor.

•    This year we saw many software announcements around OCP.  Arista announced their containerized EOS operating system (cEOS).  We saw Apstra and Cumulus active at the show as well running into many other software vendors in attendance.  OCP has done a good job at straddling the hardware/software boundary, but clearly the software needed to run these networks is quickly evolving as well.