As recent payment outages in Germany (and elsewhere) have shown, even strong proponents of a cashless economy have an interest in safeguarding the future of cash, if only for the sake of financial stability.
A few days ago, many shops in Germany had “CASH ONLY” signs displayed on their windows or at the entrance to their premises. Some establishments allowed people without cash to pay by card, but only if they provided a signature. It was as if parts of Germany’s payment system had just travelled through a wormhole back to the 1990’s, albeit with euros rather than the Deutsche Mark as the legal currency.
According to initial reports, the cause of the problems was a software glitch affecting all H5000 payment card terminals, which are widely used by German retailers including some of the largest supermarket chains. Starting on May 24 normal card payments became all but impossible for those retailers. The problems lasted for a week or so, prompting some larger retailers to replace all of their card terminals. It’s an investment that many smaller, struggling retailers can ill afford to make right now.
There are (or at least were) around 100,000 H5000 units in Germany, according to some estimates. The devices were manufactured by the US financial services provider Verifone specifically for the German market, but they are operated by eleven network operators, including Payone GmbH and Concardis GmbH.
On May 27, one of those companies, Payone, reported it was facing issues with the H5000 card terminal and that the issues were occurring throughout the country: “Like other network operators, we are currently experiencing considerable restrictions in the processing of transactions with card payment terminals of the type H5000 from the manufacturer Verifone throughout Germany.”
According to Handelsblatt, financial supervisors from Bafin and Bundesbank are already on the case. The German banking industry, which represents the interests of banks and savings banks, also announced that it would “analyze and process the disruption in depth in collaboration with the various parties involved and the supervisory authorities”. However, it also conceded that it will still take “a while” before the last H5000 terminal is replaced or updated.
The problem first came to light on May 24 when the Konsum retail chain in Dresden published the following message on its Facebook page: “Attention, an important notice for you! Due to a Germany-wide malfunction, card payments are currently not possible in our stores.”
Similar problems began to be reported by other retail chains such as Netto, Edeka, Aldi Nord, Rossman and DM as well as smaller, independent retailers and petrol stations. Initial reports suggested that H5000 card machines across Germany were experiencing a software malfunction that stopped them processing payments.
“As things stand, it will be necessary to install new software updates on all H5000 terminals, which the manufacturer will provide as soon as possible,” Payone said.
But some IT experts have pointed to a different potential cause: an expiring product certificate. The problem, it seems, is that the Verifone H5000 is a rather old albeit robust model whose so-called “End of Life” was officially announced by Verifone in 2019. The company ended its production of the terminal a year later. Although it offered limited product support until 2023, one of the embedded certificates seem to have expired unnoticed on Tuesday May 24 causing the terminals to stop working. That is the hypothesis of Jan Wildeboer, a self-described “EMEA Evengelist” at IBM-owned Red Hat who began informing the English-speaking public about the issue in a rapidly expanding Twitter thread.
Wuildeboer is a Linux expert and the H5000 was the first card terminal in Germany to run on open source software. On May 30, the Bavarian newspaper Süddeutsche Zeitung reported that a “curious detail” within the network operators’ support instructions seemed to support Wildeboer’s hypothesis:
Usually, any tech support will immediately ask the caller, “Have you tried rebooting?” In this case, the network operators advise retailers against rebooting the terminals themselves. That fits with Wildeboer’s certification thesis. Because when you reboot, the devices check their integrity, i.e. whether they may have been manipulated by hackers. If they then lack a necessary certificate, they switch to a kind of safe mode from which only an on-the-spot technician can release them.
Who’s to Blame?
Trying to apportion blame for the payment terminal outages is not going to be easy. As the article in Süddeutsche Zeitung notes, Verifone released a software update in Verifone despite having previously stated there would be no more updates after April 2021. This suggests the developers may have realized there was a potential problem with the expiring certificate. But the update seems to have reached only a few retailers, who apparently experienced no problems with their terminals. For everyone else, the devices suddenly became unusable on May 24…
Continue reading on Naked Capitalism