Performance Measures Testing
The FCC has adopted a new compliance framework to ensure that broadband service subsidized by the Connect America Fund (CAF) meets basic speed and latency performance standards.
Starting in 2020 with CAF Phase II Model participants, carriers that receive CAF support to provide fixed-location broadband service must conduct speed and latency testing of their networks and submit the results to USAC as part of the annual compliance process. As established by FCC orders in 2018 and 2019, the performance testing aims to maximize the impact of CAF investments to close the digital divide in rural America and ensure that people living in rural communities have access to the same high-quality networks as those living in urban areas.
The FCC mandates that at least 80 percent of network speed measurements be at 80 percent of required speeds and 95 percent of latency measurements be at or below 100 milliseconds round-trip time. Carriers that fail to meet the performance measures standards required of their funds may face potential withholding of support.
The performance measures testing framework requires carriers to conduct one week of speed and latency testing using a random sample of CAF-supported broadband locations with active subscribers selected by USAC in each quarter of the calendar year. Testing mandates apply to carriers participating in the following funds:
- Connect America Fund (CAF) Phase II Model
- Connect America Fund (CAF) Phase II Auction, including participants in the New NY Broadband Program
- Alternative Connect America Cost Model (ACAM), Revised ACAM and ACAM II
- CAF-Broadband Loop Support (CAF BLS)
- Rural Broadband Experiments (RBE)
- Alaska Plan
The windows to start testing vary by fund and began in 2020 with CAF Phase II Model carriers. Testing continues throughout a carrier’s support term. Carriers that participate in multiple funds must meet the separate testing requirements of each of those funds.
Carriers must test speed and latency from the premises of active subscribers to a remote test server located at, or reached by passing through, an FCC-designated Internet exchange point (IXP), which is any building, facility or location housing a public Internet gateway that has an active interface to a qualifying Internet Autonomous System (ASN). More information about acceptable test paths and remote server locations is available here. Carriers serving areas greater than 500 air miles from an FCC-designated IXP may conduct all required speed and latency testing between the customer premises and the point at which traffic is aggregated for transport to the continental U.S.
A speed test is a single measurement of download or upload speed of 10 to 15 seconds duration between a specific consumer location and specific remote server location that meets the FCC designated IXP requirements. Speed requirements vary by fund. Carriers must conduct at least one download test and one upload test during each testing hour at each testing location. (See FCC 19-104 at paras. 24-26)
A carrier may report that no test was successfully completed due to “crosstalk” caused by consumer activity if the consumer traffic meets thresholds of 64 Kbps for download tests or 32 Kbps for upload tests, and if the carrier begins attempting speed tests within the first 15 minutes of a testing hour and repeatedly retries and defers tests at one-minute intervals. (See FCC 19-104 at para. 26)
A latency test is a single measurement of latency, often performed using a single User Datagram Protocol (UDP) packet or a group of three Internet Control Message Protocol (ICMP) or UDP packets sent at essentially the same time. Carriers must offer broadband service with latency suitable for real-time applications, including voice over IP (VoIP) calling. Carriers must conduct one discrete latency measurement or observation per minute (60 per hour) for each testing hour at each subscriber test location. (See FCC 19-104 at paras. 27-38)
A carrier may postpone a latency test because of crosstalk if the consumer traffic load exceeds 64Kbps downstream.
Consumers can find more information about the FCC performance testing program here. The FCC does not require carriers to inform subscribers at locations selected for speed and latency testing because carriers already conduct routine performance measures testing of their networks.
Deployment obligations for each fund will determine when carriers participating in that fund must begin network performance testing. Test results will be due by the following July for testing conducted during the prior calendar year. USAC encourages carriers to file and certify test results on a quarterly basis after that quarter’s testing is complete.
Before the start of official testing, carriers will be subject to a “pre-testing” period for one week of each quarter of the calendar year. During pre-testing, carriers must conduct performance measures testing at a random sample of subscriber locations selected by USAC and submit and certify the results within one week of the end of the quarter, but will not face withholding of support for failing to meet speed and latency standards. USAC encourages carriers to file and certify results as soon as pre-testing is complete each quarter.
Schedule for Pre-Testing and Testing
|Program||Pre-testing Start Date||Pre-testing Results Due||Testing Start Date||Test Results Due|
|CAF Phase II||January 1, 2020||Following last week of end of each Qtr 2020||January 1, 2021||July 2022|
|Rural Broadband Experiments||January 1, 2021||Following last week of end of Qtr 2021||January 1, 2022||July 2023|
|Alaska Plan (carriers required to file in the HUBB)||January 1, 2021||Following last week of end of Qtr 2021||January 1, 2022||July 2023|
|A-CAM I||January 1, 2021||Following last week of end of Qtr 2021||January 1, 2022||July 2023|
|A-CAM I Revised||January 1, 2021||Following last week of end of Qtr 2021||January 1 2022||July 2023|
|ACAM II||January 1, 2022||Following last week of end of Qtr 2022||January 1, 2023||July 2024|
|Legacy Rate of Return (CAF BLS)||January 1, 2022||Following last week of end of Qtr 2022||January 1, 2023||July 2024|
|Alaska Plan (carriers not currently required to file in the HUBB)||January 1, 2022||Following last week of end of Qtr 2022||January 1, 2023||July 2024|
|CAF II Auction||January 1, 2022||Following last week of end of Qtr 2022||January 1, 2023||July 2024|
|NY Broadband Program||January 1, 2022||Following last week of end of Qtr 2022||January 1, 2023||July 2024|
How Testing Works
USAC is implementing the performance measures testing framework using a new system called the Performance Measures Module (PMM). The PMM: (1) lets carriers identify locations deployed with CAF support that have active subscribers; (2) generates a random sample of those locations for speed and latency testing and provides the obligated speed tiers to be tested based on the speed tiers reported for those locations in the HUBB compared with the speeds carriers are required to deliver; (3) collects the speed and latency test results from carriers; and (4) calculates compliance with performance measures standards based on certified test results.
In order to identify locations with active subscribers for testing purposes, beginning with the testing start dates and biannually thereafter, carriers must download certified locations that they have already filed in the High Cost Universal Broadband Portal (HUBB) using a comma separated variable (CSV) file and add a carrier-generated, alpha-numeric subscriber ID – such as an account number – that serves as a unique identifier for each location with an active subscriber. Carriers should not use any personally identifiable information – such as a customer phone number or social security number – in a subscriber ID. Carriers must then upload this new file into the PMM system, which will select a random sample of those locations for speed and latency testing and provide the obligated speed tiers to be tested.
Please note that for funds with multiple speed tiers, the obligated speed tier for some locations may not be the same as the speed tier reported for that location in the HUBB. That is because the system uses a cascading logic that may select locations with higher speeds to fill a sample for testing at slower speeds if there are not enough locations with slower speeds reported in the HUBB.
USAC encourages carriers to be sure that their broadband deployment information in the HUBB is as up-to-date, accurate and complete as possible before obtaining their random samples from the PMM. Carriers should first submit any new locations that have not yet been reported to the HUBB and make any necessary edits to latitude and longitude coordinates, address information and speed tier data to reflect network upgrades.
In addition, carriers should be certain that all locations with active subscribers in a study area are uploaded into the PMM before requesting a random sample of locations for testing for that study area code (SAC). Carriers should only request a random sample one time, and should not alter uploaded subscriber location data or request a new sample once a sample has been generated for a SAC. Carriers should also be certain that all locations with active subscribers in all study areas associated with the same Service Provider ID Number (SPIN) are uploaded into the PMM before requesting any random samples of locations for testing. The system will not allow carriers to obtain additional samples for any SACs associated with that SPIN once a carrier requests a sample for one SAC associated with that SPIN. The same applies to consultants who have PMM entitlements for multiple carriers.
Carriers are unable to edit the number of units for HUBB records for subscriber locations that have been randomly selected for speed and latency testing during the two years when those locations are part of the testing sample.
Because some carriers participate in funds that start pre-testing before the fund’s first deployment milestone, there may be some carriers that are supposed to begin pre-testing but have no locations yet in the HUBB. These carriers are not required to conduct pre-testing until they have deployment data in the HUBB, but should plan to begin pre-testing within one quarter after they do.
Every two years, USAC will randomly select locations for testing from among locations with active subscribers in each speed tier. Carriers must test up to 50 locations for each speed tier per state, depending on the number of subscribers a carrier has in a state per funded speed tier, and must meet separate testing requirements for each fund in which they participate. Carriers must obtain a new random sample after two years of testing/pre-testing.
Required Test Locations
|Number of Subscribers at CAF-Supported Locations per State and Service Tier Combination||Number of Test Locations|
|50 or fewer||5|
|51-500||10 percent of total subscribers|
Carriers should use the same subscriber locations for both speed and latency testing. All speed tests for the sample must be conducted in the same week and all latency tests for the sample must be conducted in the same week, but speed and latency tests may be conducted in different weeks. Carriers must conduct testing between 6 p.m. and 12 a.m. local time.
Carriers can request a replacement location if the subscriber at a location selected for testing refuses to participate, drops service or subscribes to a slower speed tier than the one being tested – leaving the carrier with fewer testing locations than required. Carriers should also request a replacement location if the subscriber at a location selected for testing does not use the service for a month or more but continues to subscribe (this would be considered “dropped service” in the PMM), or if the subscriber accepted but does not use or stopped using the modem provided for testing (this would be considered “subscriber demands removal of testing equipment” in the PMM).
If a carrier runs out of replacement locations with active subscribers at the required speed tiers, the carrier would need to upgrade service at the locations selected by USAC in order to conduct testing. Carriers that serve 50 or fewer subscribers in a state and particular service tier and cannot find five active subscribers for the required testing will be subject to verification that more subscribers are not available.
The FCC has issued a limited waiver of pre-testing sample size requirements for Original ACAM, Revised ACAM, RBE and Alaska Plan carriers in 2021 in response to difficulties that carriers have encountered installing testing equipment in subscriber homes during the Covid pandemic and obtaining the necessary testing equipment. The FCC is allowing these carriers to conduct pre-testing at only 70 percent of the USAC-generated random sample of subscriber locations in each of the four quarters of 2021.
Carriers can choose from among three options for speed and latency testing:
- A carrier may leverage existing Measuring Broadband America (MBA) testing infrastructure by using entities that manage and perform testing for the FCC MBA program to conduct network performance testing. The carrier is responsible for all costs required to implement network testing.
- A carrier may use existing network management systems and tools, ping tests and other commonly available performance measurement and network management tools to implement performance testing. This is a list of vendors that are providing services to assist carriers in conducting speed and latency testing and reporting test results to USAC.
- A carrier may develop its own self-testing configuration using software installed on subscriber gateways or in equipment attached to subscriber gateways to conduct speed and latency tests.
Carriers will submit speed and latency test results for each state and speed tier combination to USAC on an annual basis every July by uploading test results from the previous calendar year in the PMM using CSV files. Carriers will use one CSV file for speed test results and one CSV file for latency test results. Carriers must submit all test results. USAC developed this quick tips guide to the Performance Measures Module (PMM) to address frequently asked questions about how the system works and help troubleshoot common issues users may encounter.
Carriers that receive CAF support to deploy broadband may face withholding of support, after the pre-test period, for failure to meet performance measures standards. The FCC will consider failure to meet speed and latency requirements as a failure to deploy.
In order to demonstrate compliance with the performance measures standards, at least 80 percent of network speed measurements must be at 80 percent of required speeds and 95 percent of latency measurements must be at or below 100 milliseconds round-trip time.
USAC calculates carrier compliance with applicable performance standards separately for each state and speed tier – as well as each fund in which a carrier participates – and may withhold support in the event of non-compliance based on FCC guidance. Results from speed and latency testing are subject to USAC verification and audit.
For a detailed explanation of how USAC calculates compliance results, please see PMM Compliance Calculations.