Competitor Price Monitoring in the EU: What Is Actually Legal

The question I get before any other

Before anyone asks me what my crawlers can do, they ask whether they are allowed to run them. It is the right first question and most of the answers on the internet are wrong in one of two directions.

Half of them say scraping is illegal and you will be sued, which is nonsense. The other half say public data is public and you can do what you like, which is also nonsense and the more dangerous of the two, because it is the belief that gets people into actual trouble.

The honest answer is that monitoring a competitor's public prices is legal in the EU in almost every case that matters to a small or medium sized shop, and that the exceptions are specific, knowable and mostly about how you do it rather than whether you do it. This is my attempt to draw the map.

One sentence of housekeeping

I am not a lawyer and this is not legal advice. I have, however, spent years building these systems, reading the judgments that shape them and sitting in the meetings where a client's solicitor decides what they are comfortable with, and I can tell you where the lines are and which side of them I keep my work on. If your situation is unusual, and I will say below what unusual looks like, get an hour with someone who does IT law in your jurisdiction. It is the cheapest hour you will ever buy. Now I will stop saying that and get on with it.

Prices are facts, and facts belong to nobody

Everything starts here, and it is worth being precise about it because it is the point people skip.

Copyright protects works. A product description that somebody wrote can be a work. A product photo is a work. A shop's layout, its category texts, its blog, all of that can be protected and I will come back to what that means for you.

A price is not a work. Neither is a stock level, a delivery time, a product name or an EAN. They are facts about the world, and the entire tradition of European copyright law, going back long before the internet, says that facts are free. You cannot own the fact that your competitor sells a particular coffee machine for 349 euros any more than you can own the fact that it rained in Vienna on Tuesday.

This matters because it removes the most intuitive objection immediately. When a competitor says "you cannot use our prices", the first answer is: those are not yours in the way you think they are. What they might have is a right in the collection of prices, or a contract with you about how you use their website, or a competition law claim about your conduct. Those are the real arguments and I will take them one at a time. But the price itself, as a fact, is nobody's property.

The database right, and why a shop's own prices are usually outside it

The EU has a peculiar piece of law that exists almost nowhere else. The Database Directive of 1996 created what is called the sui generis database right. It protects a collection of data, not because the data is creative, but because somebody invested substantially in obtaining, verifying or presenting it. Extract or reuse a substantial part of such a database without permission and you infringe, even if every individual item in it is a plain fact.

This is the argument that scraping opponents reach for first, and it is weaker than it looks, for two reasons.

The first is a decision from 2004 in a case brought by the British Horseracing Board against William Hill. The court that decides these things for the whole EU held that the investment which earns protection has to be investment in obtaining existing data, not in creating it. A horseracing authority that decides which horses run in which race is creating that data as a by product of running the sport, and the court said that does not count. The same logic applies rather neatly to a shop. Your competitor did not go out and collect their prices from the world. They decided them. The price list is a by product of running a shop, and on the reasoning of that case, the investment in deciding prices is not the kind of investment the database right rewards. Lawyers call this the spin off doctrine and it is not universally applied with the same enthusiasm in every member state, but it is the starting point.

The second reason is the word substantial. The right is infringed by extracting a substantial part of the database, judged by quantity or by quality, or by repeatedly and systematically extracting insubstantial parts in a way that amounts to the same thing. Monitoring the prices of the forty products you actually compete on, out of a competitor's catalogue of six thousand, is not a substantial part by any reading. Mirroring their entire catalogue every hour is a different conversation, and this is the first of many places where the answer depends on how you do it rather than whether you do it.

There was also a case in 2013 about a dedicated search engine that queried a car sales website in real time and re presented the results, and the court found that infringed, because the search engine was effectively offering the public the whole database through a different front door. That is a useful boundary marker. Monitoring for your own commercial decisions sits on one side of it. Republishing a competitor's catalogue as your own service sits on the other.

Terms of service: browsewrap, clickwrap and the Ryanair cases

If the database right does not protect the prices, the next thing a competitor reaches for is the terms of service. Somewhere on their site, in a footer link nobody clicks, is a paragraph saying automated access is prohibited. Does that bind you?

The honest answer is: it might, and the case law that tells us so comes almost entirely from one airline.

Ryanair has spent the best part of two decades suing anyone who scrapes its fares, mostly flight comparison sites and online travel agents, in courts across Europe. Two decisions matter for us.

In 2015 the EU's top court decided a Ryanair case against a Dutch comparison site and held that where a database is not protected by copyright or by the database right, the owner is free to restrict its use by contract. In other words, the fact that the database right does not cover the data does not stop the terms of service from covering it. This is the decision that scraping opponents quote, and it is real.

But a year earlier, in 2014, the German Federal Court of Justice had decided a Ryanair case against a German flight broker, and the finding there was that using a website against its terms is not, on its own, an act of unfair competition. Breaching a contract you never clearly agreed to is not the same as unlawfully obstructing a competitor. What the court did flag as potentially different was the circumvention of technical protective measures. Getting around a barrier the site put up to stop you is where the analysis changes.

Put those together and you get the position most practitioners work from today. Terms can matter, particularly if you actively accepted them, which is what clickwrap means: you created an account, ticked a box, agreed. A footer link that you never saw, which is browsewrap, is much weaker, and in Germany and Austria in particular it is doubtful that a mere visit to a public page creates a contract at all. And even where terms bind, the remedy is a contractual one, not a finding that you competed unfairly.

I will say what that means in practice in a moment, but the short version is: I do not create accounts on competitor sites to crawl them, and I do not crawl anything that requires agreeing to terms to see.

The line that actually matters: logins and technical barriers

Almost every scraping case that ended badly for the scraper has one of two features. Either the data was behind a login, or the scraper defeated something that was put there to stop it.

The login point is straightforward. If you have to authenticate to see the data, then the data is not public, you have almost certainly clicked to accept terms, and you are inside somebody's system on conditions. Germany's criminal code has a provision, paragraph 202a, about accessing data that is specially secured against unauthorised access by circumventing that security. Austria has a close equivalent. The UK has the Computer Misuse Act. None of these were written with price monitoring in mind, but they were written broadly, and a prosecutor does not need to prove you intended harm.

The technical barrier point is subtler and is where reasonable people argue. If a site rate limits you and you back off, you are being a good citizen. If it blocks your IP and you rotate through a residential proxy network to get around the block, you have just done the thing the German court in 2014 flagged as the line. If it puts up a CAPTCHA and you pay a service to solve it, same. The site has said, in the only language a server speaks, that it does not want this, and you have overruled it.

My rule is simple. Public pages, no login, and if the site pushes back I stop rather than push harder. That rule costs me some data. It has also never cost a client a letter from a lawyer.

Unfair competition: UWG in Germany and Austria

Germany and Austria both have an unfair competition act, in each case called the UWG, and it is the law that competitors actually use against each other. The relevant concept is targeted obstruction: conduct whose purpose or effect is to hinder a competitor rather than to compete on merit.

The 2014 Ryanair decision I mentioned is the anchor here, and it is reassuring. Collecting publicly available prices to inform your own commercial decisions is competing on merit. It is, in fact, what competition is.

Where the UWG bites is when the crawling itself does damage. Hammering a competitor's server hard enough to slow it down for real customers is obstruction. Crawling so aggressively that they have to pay for more infrastructure is obstruction. Collecting their content and republishing it under your name is a separate wrong. And there is an old line of cases about systematically undercutting a specific competitor with the sole aim of driving them out, which is obstruction regardless of how you found out their prices.

None of that is about the monitoring. It is about what you do with it and how much load you cause. Both of those are fully within your control.

Text and data mining, the DSM Directive and the opt out

In 2019 the EU passed the Copyright in the Digital Single Market Directive, which among other things created two exceptions for text and data mining. One is for research organisations and does not concern us. The other allows anyone to make copies of lawfully accessible works for the purpose of text and data mining, unless the rights holder has expressly reserved that use, and for online content the reservation has to be made in a machine readable way. Germany implemented this as paragraph 44b of its copyright act and Austria as paragraph 42h.

Why does this matter if prices are facts and not works? Because in practice a crawler does not fetch a price in isolation. It fetches a page, and the page contains a product description, images, structured data and category copy, all of which can be works. The TDM exception is what makes the copying of the page lawful for the purpose of extracting the facts from it, provided you are not overriding an opt out.

The opt out is where the action has moved in the last two years, mostly because of AI training rather than price monitoring. Sites are adding machine readable reservations in robots.txt and in dedicated TDM headers, and the emerging consensus is that a properly expressed reservation has to be respected. My crawlers check for these and honour them. I also keep the copies for as long as it takes to extract the facts and no longer, which is what the exception contemplates, and I do not store or republish the descriptions and images.

GDPR: where product data stops and personal data starts

A price is not personal data. Neither is a product, a category, a shipping rate or a stock level. For most of what a price monitor collects, the GDPR is simply not engaged, and anyone who tells you otherwise is either confused or selling a compliance product.

But a crawler that is not careful will pick up personal data without meaning to, and there are three places it happens.

Reviews. A product page has reviews, the reviews have names, sometimes towns, sometimes photos. The moment your crawler stores "Anna M. from Graz thinks this blender is loud", you are processing personal data of somebody who has no idea you exist, and the GDPR says you would owe her information about that within a month. Nobody does this, which means everybody doing it is non compliant. I do not collect reviews. If a client wants sentiment, we look at aggregate ratings only.

Sole traders. In Germany and Austria a huge number of small shops are run by individuals under their own name. "Max Mustermann e.K." is a business and also a person, and the imprint, the bank details and often the address are personal data. My crawlers do not touch imprint or contact pages and do not store merchant identity beyond a domain name.

Marketplace sellers. On Amazon or eBay the seller name on a listing is often a person. Monitoring a listing's price is fine. Building a profile of who sells what is not, at least not without a legal basis you can articulate.

The pattern is the same as everywhere else in this article: the data you actually want is unproblematic, and the trouble comes from collecting more than you want because it was on the same page.

Marketplaces are a different country

Amazon, eBay, Zalando, Otto and the rest deserve their own paragraph because the terms are different and so is the enforcement.

Every large marketplace prohibits automated access in its terms, and unlike a small shop's browsewrap, those terms are ones you have almost certainly accepted, because you sell there, and your seller agreement binds you even when you are crawling as a buyer. Amazon in particular is technically sophisticated about detecting crawlers and commercially aggressive about suspending seller accounts. A suspended Amazon account is an existential problem for many of my clients and is not worth any amount of pricing data.

So for marketplaces I take a different approach. Where there is an official API, and Amazon's Selling Partner API and Product Advertising API both exist for this reason, I use it, on its terms, with the rate limits it imposes. Where there is not, I use a licensed data provider who has their own arrangement with the platform and who carries that risk. What I do not do is point a crawler at a marketplace on a client's behalf and hope.

That is not a legal judgement so much as a commercial one. The marketplace has all the leverage and the client has none.

Everybody does it, including the people telling you not to

It is worth stepping back and noticing what the market actually does, because it puts the risk in proportion.

Amazon monitors competitor prices continuously and has done since the beginning. It is the entire basis of its pricing engine. Every large electronics retailer in Germany runs price monitoring against every other one, often through the same handful of specialist providers. Manufacturers monitor their retailers. Retailers monitor their manufacturers' direct channels. Comparison sites exist entirely on crawled data and have for twenty years. There is a whole industry of Amazon price history trackers used by millions of consumers.

If public price monitoring were illegal, the entire European retail sector would be in breach. It is not illegal. What has happened, over twenty years of litigation mostly funded by an airline, is that the edges have been defined. Public pages, no circumvention, no republishing, no personal data, no damage. Inside those edges is where everybody operates, and it is a large space.

The competition law trap nobody warns you about

Now the risk that actually keeps me up, and that almost no scraping guide mentions because it has nothing to do with scraping.

Article 101 of the EU treaty prohibits agreements and concerted practices between competitors that restrict competition, and price coordination is the most serious category. The fines are a percentage of global turnover. The German and Austrian competition authorities enforce this vigorously.

Monitoring your competitors' public prices and reacting to them is not an agreement. It is the opposite of an agreement. It is what competition looks like, and every competition authority in Europe has said so.

But here is how it goes wrong.

Sharing the data. Two competitors who both subscribe to the same monitoring and then compare notes about what they see have moved from monitoring to exchange. If the exchange lets them anticipate each other's moves, that is a concerted practice.

Signalling. Setting your price to send a message rather than to sell, in the expectation that the competitor's monitoring will pick it up and respond, is a recognised form of tacit coordination. Regulators have written about algorithmic collusion for a decade and the concern is exactly this: two repricing bots that learn to stop undercutting each other.

Resale price maintenance. This one is for manufacturers and brands, and it is where enforcement has actually landed. In 2018 the European Commission fined four consumer electronics manufacturers over a hundred million euros between them for using price monitoring software to detect retailers who discounted below the recommended price, and then pressuring them to raise it. The monitoring was legal. Using it to enforce a minimum resale price was not. If you are a brand monitoring your own retailers, this is the paragraph to read twice.

Hub and spoke. A third party, say a monitoring provider or a marketplace, that passes pricing intentions between competitors can make them all liable even though they never spoke to each other directly. There was a case about a travel booking platform that sent a message to all its agents about capping discounts, and the agents who did not object were held to have participated.

The rule I give clients is: your monitoring data goes into your pricing decisions and nowhere else. It does not go to a trade association, a WhatsApp group of friendly competitors, or a manufacturer who asks nicely. And your repricing rules should be written so that a human can explain why each price is what it is, in terms of your own costs and strategy, not in terms of what a competitor will do next.

Good citizenship as evidence

Everything in the last few sections comes down to conduct, and conduct is easier to defend when it is documented. A few things I do on every crawler that cost nothing and change the conversation if there ever is one.

It identifies itself. My crawlers send a user agent that names the operator and links to a page explaining what the crawler does and how to contact us. A site owner who wants us to stop can ask, and we stop. That is a very different posture from a crawler pretending to be Chrome on a residential connection.

It reads robots.txt and means it. Disallowed paths are not fetched. Crawl delay directives are honoured. TDM reservations are honoured.

It is slow. A price does not change every second. Fetching a product page once a day, or a few times a day for volatile categories, from a single IP, with polite gaps between requests, produces all the data anyone needs and puts less load on the target than one curious human with a browser. There is no legitimate monitoring purpose that needs to hammer a small shop.

It backs off. A 429, a 503, a block: the crawler waits and retries later, and if it keeps happening it stops and tells me. It does not rotate proxies. It does not solve CAPTCHAs.

It has a written purpose. One page, in plain language, saying what is collected, from whom, why, how long it is kept and who can see it. This is also most of a GDPR record of processing, if one turns out to be needed, and it is the document you hand to a solicitor when a letter arrives.

I think of these less as compliance and more as the difference between a professional and a hobbyist. It also happens to be what a court looks at when deciding whether you obstructed anyone.

What a compliant crawler looks like

Putting it together, here is the specification I build to. It is the one under GrowCentric's competitive insight, and it is the one I would insist on for any client.

Public pages only, no accounts, no logins, no paywalls. Honest user agent with a contact. Robots.txt, crawl delay and TDM opt outs respected. Rate limited to a fraction of what a human browser would generate, from identified IPs. Product level data only: name, identifiers, price, availability, shipping cost, variant attributes. No reviews, no reviewer names, no imprint data, no seller profiles. Page copies discarded once the facts are extracted. Descriptions and images not stored or reused. Retention of the extracted facts limited to what the analysis needs, typically a rolling window, and documented. No marketplace crawling; APIs or licensed providers instead. Monitoring output stays inside the client's own pricing process and is never shared with competitors or used to enforce resale prices. A one page purpose document, reviewed when the scope changes.

That is a longer list than "just scrape it", and it is also the reason I can build this for regulated clients who would not touch a generic scraping tool.

What I will and will not crawl for a client

Since prospects ask, here is the practical version.

I will monitor a competitor's public product pages and category listings for price, availability, shipping and promotions. I will monitor a manufacturer's direct channel if you are a retailer, and a retailer's public prices if you are a brand, with the competition law caveat above hammered into the brief. I will monitor price comparison sites, which are themselves aggregators of public data. I will track your own listings across marketplaces through the marketplace's own API.

I will not create accounts or crawl behind a login. I will not defeat blocks, rotate proxies or solve CAPTCHAs. I will not crawl marketplaces directly. I will not collect reviews, reviewer identities, imprint pages or seller profiles. I will not store or republish product descriptions or images. I will not build anything whose output is designed to be shared with a competitor or used to police a retailer's prices. And I will not crawl a site that has asked us to stop.

If a client wants something on the second list, that is fine, we simply do not work together on that piece. It has happened twice. Both times the client later thanked me.

The checklist

If you are evaluating a monitoring setup, yours or a vendor's, ask these questions. A good answer to all of them and you are inside the space where the whole European retail sector operates. A bad answer to any of them and you have found your risk.

1. Is everything collected from public pages without a login?
2. Does the crawler identify itself and provide a contact?
3. Does it respect robots.txt, crawl delays and TDM opt outs?
4. Is the request rate low enough that the target would not notice?
5. Does it stop when blocked rather than route around the block?
6. Is the data limited to product and price facts, with no reviews, names or contact details?
7. Are page copies discarded and descriptions and images left alone?
8. Are marketplaces handled through APIs or licensed data, not direct crawling?
9. Is there a written purpose document, and does it match what the system actually does?
10. Does the output stay inside your own pricing decisions and never reach a competitor or get used to enforce resale prices?

If you can say yes to all ten, the question you started with, are we allowed to, has a boring answer. Yes. Now the interesting question is what to do with the data, and I have written about that in beating price wars in homogeneous product markets and in game theory in growth strategy. The next piece in this series is about the repricing rules themselves, and how to use competitor data without starting a price war you cannot win.