Spirent circle logo

The Importance of Conformance and Performance Testing


Blog - The Importance of Conformance and Performance Testing

Most projects or product developments are rarely on-time or on-budget. The culprit is usually a breakdown in communication and a lack of mutual understanding of what is being planned, defined or even recommended.

Target Clock Money

Most projects or product developments are rarely on-time or on-budget. The culprit is usually a breakdown in communication and a lack of mutual understanding of what is being planned, defined or even recommended.

In the worst-case scenario, the finished product has not been tested thoroughly. Imagine you are a car manufacturer and plan on developing a new infotainment platform. You have several suppliers delivering into your project, and you’re under immense pressure, due to a strict deadline and a limited budget. The best workaround in this case would be to either try to get some things for free or go the open source route. Your suppliers are thinking the same thing and will also utilize free and openly available platforms. They need to test if their product works, which helps save money on your end. This might work, but in most cases, you will run into problems by not addressing issues from the beginning.

Layer 1 – Starting from the Ground Up

All structures need a good foundation, especially for high speed, high bandwidth applications like Ethernet. The physical layer or PHY, is one of the most important layers to get correct because everything must go through this layer. It consists of many components and functions including Physical Media Attachment (PMA), Physical coding sublayer (PCS), and PHY register lookup and control, etc.

Designing and developing printed circuit boards with common mode chokes (CMC) for usage with the wide range of environmental, thermal, electrical vibration and shock conditions within an automotive network is a large task. Taking the Electromagnetic Compatibility (EMC) standard into consideration also adds another layer of complexity. If you are used to designing high speed networking equipment for IT data centers or switches, which reside in air-conditioned controlled environments, the automotive thermal envelope is a whole new world.

Bits, Bytes, Sweat and Tears

First Aid

Don’t lose sleep over your physical layer! It resides in the world of physics, by having the right test equipment, testing and validating the physical layer becomes much easier. Having the right partners can help ensure that you meet your design needs and overcome any obstacles. This is exactly why Spirent partners with many industry leaders who offer the correct services to help customers excel at developing not just one, but multiple products in Automotive for speeds up to 10G.

With every high-speed design, there will be some learning curve. Customers are currently cutting their teeth learning about infotainment and multi-media systems. How are your designers learning Ethernet development? Where are your lessons learned? It is always cheaper, in terms of time and budget, to find and fix the bugs early in the development cycle by having the right tools at hand.

Ethernet is becoming the backbone of vehicle functions, so its operation needs to be seamless and reliable to meet future demands for safety.

Complacent for Compliance is a Show Stopper

The communications interface is the only way to get information into and out of a system. However, if a system is not interoperable, meaning if vendor A’s bit and bytes implementation is not synced up with vendor B’s, C’s and D’s bit and byte implementation, each system will not be able to understand or translate the correct meaning of the bits and bytes received. This is a big issue and is the root for conformance testing for interoperability.

There is no middle ground when it comes to a safety critical software release. The entire network needs to be functioning properly 100% of the time under all circumstances, under all voltage conditions, environments and temperature ranges when deployed. Testing takes on a whole new meaning for Automotive Ethernet, with fierce competition and new vendors popping up in the equipment vendor market place every day.

Product Differentiation – The Proven Pedigree

How are you differentiating your product from the competition? I advise my customers to hand an Original Equipment Manufacturer (OEM) a conformance test suite report stating a baseline for their Software implementation. This will give the OEM confidence & help prove that their vendor or supplier has the right tools to carry along their program on-time and on-budget. This can only be done by having the right tools and partners in place, early in the requirements and design concept stages.

When looking at Ethernet based communications, we need to define exactly which standards to implement inside the network. These standards need to take into consideration which IEEE802.1as (aka gPTP) timing profile you are planning to use. If your devices are not speaking the same gPTP profile, they will not synchronize.

The specification selections can be simplified depending on your application needs. For example, using AUTOSAR conformance test suites for embedded designs where customers have some type of powertrain or propulsion involved, or using OPEN Alliance TC8 for more dynamic configurations. Avnu Automotive profiles are also used in embedded designs, they cover additional sections like startup, diagnostics, exceptions and timing.

Common Ethernet Benchmarking Methodologies for Ethernet Communications - RFC

When customers want to benchmark their switch performance, they typically use a proven methodology to ensure the way vendors instrument the measurement process is proven and accepted by governing bodies like the IETF (Internet Engineering Task Force).

Spirent pioneered the RFC2544 and RFC2889 methodology with the IETF around 20 years ago and continues to provide this capability with nanosecond accuracies. Below list contains key test cases.

RFC2544 Benchmarking:

  • Throughput

  • Latency (LIFO, FIFO, LILO) for Cut-through and store and forward devices

  • Frame loss

  • Back-to-back frames

RFC2889 Benchmarking:

  • Forward rates, with 10ns resolutions

  • Congestion control

  • Address caching

  • Address learning

  • Error filtering

  • Broadcast forwarding

  • Broadcast latency

  • Forwarding pressure

  • Maximum forwarding rate

Don’t Leave Noise Testing to Chance

It’s a noisy world out there. As Ethernet speeds increase from 100Mb/s to 1000Mb/s, noise testing is even more critical to application performance, as packet sizes are smaller and more susceptible to noise phenomena at these higher speeds.

So, what is noise testing? It can be interpreted in many ways. I define it as the ability of a receiving system to reconstruct critical signal flows with missing packets in the middle of the sequences. After spending a decade developing and testing network communications, it is clear to me that if you are not testing how your applications react to a noise phenomenon or how it is to recover after several noise events, it could spell trouble for the future. It is way too late to uncover a noise recovery issue during the vehicle testing stage. Finding a way to resolve them at that stage may prove to be very costly. As we see more and more electronics being introduced in vehicles, noise may come from different sources in or outside the vehicle. Spirent has a unique and patented way to inject noise in a repeatable manner to provide users the ability to see everything from a single packet drop to complete link failure.

Finding Problems Early

Conformance Testing is also called positive testing. Testing for conformance, helps assure that the product under test relies on certain protocol standards, and that they are correctly implemented based on industry standards requested by the OEM.

Besides Protocol Conformance Testing, which is a positive test, there is also a negative test, which covers unexpected communication failures. Combining these techniques in a regression test covers most possible test cases. By combining performance and noise testing, many hidden issues, known or unknown, may come to light.

Testing in general is not just about accuracy, but also about coverage. You simply can't test everything. However, testing helps ensure that suppliers are providing the correct implementation and conformance to the standards specified. The best solution would be to ask them for a certificate from an accredited test lab, or a test reports showing what is working and what is not, so a baseline may be developed for a specific software release along with aspects to improve on in the future.

Finding problems early is always cheaper than finding them later. Below is an illustration of costs associated with finding defects. It is 15x times costlier to fix a bug in the testing phase than in the designing phase.

Relative Cost of Fixing Defects

If you could have two of the three items below, which would you choose?

Blog image

Read press release on new partnership announcement delivering complete test coverage for all layers of Automotive Ethernet conformance.

Like our content?

Subscribe to our blogs here.

Blog Newsletter Subscription

Jeff Warra
Jeff Warra

As a Business Development Engineering at Spirent, Mr. Warra works on developing the next advancements in the connected vehicle landscape for the automotive, aerospace and off-highway sectors. Jeff has over 18 years of experience in the engineering field at Tier 1, OEM and test equipment manufacturers. Jeff has become a specialist on advanced technologies and has held various positions in the industry from development, test, applications and project engineering. Being focused on safety critical systems early in his career has allowed him to build a solid foundation on electrical and software engineering principles and practices.