Data mining and AI through the ages - from horse-drawn vehicles to supersonic aircraft: An interview with Dr. Michael Thess

Dr. Michael Thess – mathematician, visionary and co-founder of the prudsys AG. He has shaped the company and its software solutions like few others. From 1998 to 2016, Dr. Thess was one of the managing directors responsible for algorithm and product development and still maintains close ties to prudsys AG today. But what only few people know is, he is the silent founder of the prudsys solution for dynamic price optimization – today one of the world’s most powerful software products in the field of dynamic pricing (see GARTNER report) and certified Solution Extension of SAP. 

“The road to get here was arduous,” Dr. Thess told Mrs. Pricing in an interview. “It was tough! We struggled.” he recalled. “We wanted to support and optimize retail processes with intelligent data analysis. However, when it comes to data analytics and stable predictions, retail is not necessarily benign. By its very nature, it is very volatile,” Dr. Thess summarizes with a smile. But he knew one thing: “Price optimization – that’s at the heart of every retailer. If price optimization doesn’t work, every retailer has a problem. In that sense, I didn’t invent price optimization per se, but I recognized the potential right away. That’s why, above all, I pioneered the prudsys dynamic pricing solution.”  

This gives us reason to look back and see how it all started. The beginnings of prudsys AG go back to 1993, the year in which Jens Scholz (now CEO of prudsys AG) and Dr. Thess founded their first joint company, Scholz & Thess Software GbR. Their product was a data mining tool for analyzing small amounts of data. At the end of the 90s, the internet was spreading rapidly and all companies were suddenly focusing on digitization. “It was clear to us then that if data was flowing together from a wide variety of sources, then the next step was to be able to analyze these huge volumes of data in a meaningful way,” Dr. Thess recalled. This could not be done with the comparatively small data mining tool. Therefore, the GbR founders looked for other partners with whom they could think bigger about data mining, and founded the prudsys AG in 1998. Strategically and technically, the main focus of the prudsys AG since then has been on two essential aspects that are decisive for success in cross-channel commerce 

  • Bringing together different data sources and making them usable (keywords: omnichannel and multi-client capability), 
  • Processing and evaluating enormously large data volumes intelligently and with high performance. 

“In other words: We had no lesser goal than to overcome ‘the curse of dimensions.’ When we started, you could only compute and resolve a maximum of three dimensions in a meaningful way – that was the state of the art at the time. But three dimensions were far too few because customer service in retail needs at least five dimensions: the customer, the right offer, at the right time, via the right channel, and at the right place,” said Dr. Thess. Support Vector Machines (SVMs) were commonly used for this purpose, which was highly innovative at the time. The problem with these methods, however, was that they could not support much more than three dimensions. When computing higher-dimensional tasks, SVM-based methods reached the limits of what was technically possible and economically viable in terms of the computing power required. In his search for a high-performance solution, Dr. Thess found a promising alternative in sparse grids methods. Sparse grids methods work on attributes (instead of vectors) and can intelligently and skillfully combine large amounts of data points. This approach not only allows the processing of several vectors within one attribute at the same time, but also includes a multitude of dimensions. In this way, complex trading processes can be mapped mathematically and optimized in a targeted manner. “With this, we were finally able to dissolve ‘the curse of dimensions.’ That was one of the biggest breakthroughs of my life,” Dr. Thess noted.  

 “When we then began to earn a name for ourselves in the field of recommendation engines in the early 2000s, we took the next step in our technological evolution,” he explained further. “Because in order to be able to do data mining not only in a performant way, but above all in a goal-oriented way, we still needed an important building block: intelligence. That’s how we came to Reinforcement Learning (RL). This technology has taken our software to a whole new level.” Reinforcement Learning leads to:  

  • Real-time performance and mass data processing 
  • The artificial system is able to learn, to adapt to its environment in a fully automated way and to evolve autonomously based on a predefined goal 
  • Goals can be specified by the user according to business strategy, and the artificial intelligence (AI) can be controlled specifically with regard to margin, turnover or sales optimization. 

 “That was a revolution! And ultimately, it was also Reinforcement Learning (RL) that brought us to Dynamic Pricing. Through RL, we had to connect everything in real time. It’s important to understand what that means: we now had all the relevant data, in real time. And we knew it could do a lot more than ‘just’ product recommendations. That’s how we finally came to Dynamic Pricing. I had always been interested in price optimization. I was also encouraged by our Recommendations customers who asked us, as data miners, for a solution for dynamic price adjustments. That was the logical next step. If we could provide product and content recommendations in real time – why not provide prices in real time? This idea manifested itself in 2006,” Dr. Thess recounted.  

 “In the same year, I started to develop the first algorithms. At that time, it was still on a very simple level. The first pricing algorithm could do one thing above all: maximize revenues. But that’s hardly the only way to inspire a category manager – that’s for sure,” Dr. Thess noted with a laugh. “In close cooperation with our customers and our prudsys team, the methods were then further developed. At first, we focused on the fact that the algorithms were primarily margin-optimizing. Ultimately, however, it was our customers who made us realize that the category or pricing manager must be able to determine for themself which KPIs our procedures will optimize. So we started with an intuition, and step by step one thing led to another,” he continued. So, almost everything that has been achieved since 1993 pays off in the great strengths of the current AIR solutions: 

  • Reinforcement Learning in real-time, 
  • Linear scalability and performance with enormous amounts of data, 
  • Combination of personalization and pricing. 

 “When I look back, we’ve gone through an interesting evolution since our beginnings in 1993: sort of from a horse-drawn vehicle to a rugged truck to a high-performance aircraft. Our horse-drawn vehicle was the simple data mining approaches of the mid-1990s. Then came the sparse grids, which were already a strong truck in comparison. And finally, the supersonic airplane, or Reinforcement Learning technology, which we adapted for trading processes and turned into a competitive advantage for retailers.”  

 Now, we’ve discussed a lot of the company’s history, from the internet boom and the technological evolution from horse-drawn vehicles to supersonic aircraft. That almost automatically begs the question of looking into the future. So I asked Dr. Thess where he thinks technological development will go and what trends he predicts for the coming years: “I think reinforcement learning will remain the topic of the future for retail – both stationary and online. I see great potential in implementing reinforcement learning via hierarchical tensors. This is because tensors conserve resources and are simply perfect for processing mass data. This opens up very great potential for omnichannel commerce, which is becoming more and more standard.”  

 Thank you, Michael! Not only for this interview 😊 

Mrs Pricing