Can a machine read a review profile? Fourteen hosts, measured

You can read a review profile. You just cannot always read the rules first.

The received wisdom on review platforms is that they block machine reading of product pages, usually with a line about their terms of service. Measured against the access rules themselves, that is not what we found. Of the 14 hosts we probed, 12 publish a rules file, and in 11 cases the rules that apply to all crawlers permit reading a product profile. The rules are aimed at something else: following a vendor, filing a lead, posting a review, hitting an API.

The obstacle is the door, not the floor plan. 2 of the 12 hosts that publish a file serve it to a browser and a bot wall to a machine, so an automated client cannot read the policy that governs it. 9 answer a 403, 1 answers a 429, 1 refuses every identity we tried, and 1 does not resolve at all.

That distinction changes what a marketer can honestly claim. A 403 is not a policy you can quote, because there is no policy in front of you. It is a wall, and the honest reading of a wall is that the site is closed to you, not that it has published a condition you have failed.

The practical answer for a buyer who needs a citation is narrow but real. Where the rules are published and they permit reading, you can read, and you can say so. Where they are not published, you cannot cite the platform's policy, because there is not one you have been given. What you can do instead is ask the vendor, keep your own measurements, and publish those.

Last verified 2026-10-23. Access rules change without notice, and two of the fourteen hosts answered differently depending on who asked. Re-read the table before you rely on a row in it.

What each host served when a machine asked for its rules

One request for the rules file, as an unconfigured client and as a browser, on 23 October 2026. The plain request was repeated hours later and returned the same answer on every host, so the refusals below are repeatable rather than a bad afternoon. No product page was fetched from any host on this page.

How a machine was answered, across the fourteen hosts

Every host gave one of five answers, and the five mean different things to a buyer.

The blue segment is the one that matters most for a buyer: those hosts have a published policy, and a machine is not allowed to see it.

Four different kinds of no, and they do not mean the same thing

Three things get called a refusal, and there is a fourth that is not one. They lead to three different decisions, and a buyer who treats them the same will either give up on a site that would have served them, or keep pushing at one that was never going to answer.

**A 403 with a bot-mitigation page.** The host serves a challenge page instead of a rules file. Nothing has been published about what an automated client may read, because the request never reached the rules. 9 of the hosts answer this way to a machine while serving a real rules file to a browser, which is what makes it a wall rather than a policy.

**A 403 with no rules file under any identity.** The host serves the same challenge page to every identity we tried. There is no policy to read and no version of the question that gets an answer. 1 host in our sample behaves this way, and it is the only one where we can say the site is closed to automated reading as a matter of published intent rather than of infrastructure.

**A 429.** A rate limit, which is about volume rather than identity. It tells you nothing about permission and everything about how slowly to go. One host answered this way, and it also serves a rules file to a browser, so a slow retry would have got us the policy.

**A host that does not resolve.** A third category again, and it is not a refusal at all. 1 of our hosts did not resolve on any attempt. Nothing has been decided about us, because there is no server to decide anything.

What the published rules actually block

The rules that apply to all crawlers were read on every host that served a file, then applied to representative product profile paths: a product page, a review listing, a category page. The split that matters is between rules that stop you reading a page and rules that stop you acting on one.

The rules are aimed at the things a crawler would do, not the things it would read

Once you read the rules rather than the folklore about them, a pattern appears across the hosts that publish them, and it is consistent enough to be a design decision rather than an accident.

The action rules cluster on a small number of path shapes. Following a vendor and being counted as a lead. Submitting a review or a rating. Posting a comment or a discussion. Calling an API. Query-string forms that would let a crawler walk a filtered view of a database. Across the hosts that published a file we counted 64 rules of that kind against 96 that target page content.

That is the right way round, and it is worth saying so plainly, because the folk belief has it backwards. A review platform wants its ratings to exist, wants vendors to read the comparison pages their customers are reading, and wants the pages indexed. What it does not want is the rating form submitted ten thousand times, or a lead booked for a sales team that did not sell anything.

The read rules that do exist are mostly narrow. One host blocks query strings that look like a search interface, another blocks a printable or exportable variant of a page, another blocks a preview mode. These are the paths that turn a crawl into a database dump rather than a reader. A buyer reading a product profile is doing the thing the rules mostly permit.

How much each host publishes for all crawlers

The number of rules a host publishes for every crawler, against the same hosts' answer to a machine. It is the obvious thing to check first, and it settles nothing.

One of the hosts in this sample publishes 88 rules for all crawlers and permits reading a product profile. Another publishes 2 rules and permits the same. The size of the file says nothing about whether you can read.

Each host, with what we could read

The hosts that published a rules file, one at a time, with the rules that apply to all crawlers and what they do. This is the part you would otherwise have to assemble by hand, which is exactly why the folklore about this category is so confident and so wrong.

One host publishes a file that closes the whole site to you

The rules we read for this study are on file, and the most consequential line in them belongs to a single host.

Most review platforms publish a file with a rules group for every crawler and a longer, more specific group for the crawlers they want. One host in our sample publishes a group for all crawlers that reads, in full, Disallow: / . Everything else on that host is reachable only to a named list of search crawlers. Exactly 1 of the 12 hosts that published a file closes the whole site this way.

That is a published refusal rather than a missing file, and it is the one case in this sample where you can point at a line of text and say the site has said no to automated reading. It is also the reason we checked the file for an HTML body before reading it: a challenge page served with a 200 looks, to a script that only checks the status code, like permission.

There is an irony worth naming. The same host keeps a detailed rules group for named crawlers, with 34 rules that permit reading reviews while blocking the evaluation and submission paths. In other words, the policy is not that reviews are private. The policy is that the reading is for the search engines.

What actually happens when you try to read one profile

The abstract version of this study is easy to nod along to and then get wrong. Here is the sequence, on one host, with the requests that were made.

You want a product profile from a review platform. You read the platform's rules first, as you should, and the host answers with a 403 and a challenge page. There is no rules file. You do not have a policy, and you do not have permission, and the two are not the same thing to give up.

What you do have, on this host, is a published file, because a browser gets one. The file permits reading a product profile and blocks the action paths. So you know what the policy says, and you know you are not the client it was written for.

Here is where most of this goes wrong in practice. A rules file reached by pretending to be a browser is not a permission granted to you. The host has not given an automated client access, and reading the file that way does not change that. What the file is good for is knowing what the policy would be if you were entitled to it, and therefore knowing that a colleague with a browser and a licence to use the data should be asking the vendor for the licensed route rather than for the crawl.

The defensible position is short. Read the rules you can get. Respect the wall. Ask the vendor for the data under an agreement, or cite their own published claim, or use your own measurement. What you must not do is treat a challenge page as a policy, because the moment you write that in a research note, someone else will read it as a permission that was granted.

What to do when you cannot read a profile

The answer is not to find a way in. It is to change what you go looking for.

What we do, stated plainly

This study is also the rule we have been working under, written down so that it can be checked against us.

We read a site's rules before we read anything else on it, for the exact path we intend to take. Where the rules are clear, we follow them, and we keep the file with its date and hash so a reader can re-check our reading. Where a host serves a bot wall instead of a rules file, we treat the site as closed and we do not fetch its pages. We do not switch identity to get around a wall, and we do not use a headless browser to get around one either.

For this study we did make a request under a browser identity, and we want to be exact about why. We made it to characterise the response, not to obtain content we had been refused, and the only thing we fetched under that identity was the rules file itself. No product page was fetched from any host in this study, under any identity, at any point. If you are reading this to work out how to read G2 product pages, we have to be direct: the answer in this data is that you should not, and the reason is the wall, not the rules.

The six questions this study keeps getting asked

Four readers, and what each of them should take

The same measurement reads differently depending on what you are doing with it.

**You are a marketer who needs a citation from a review platform.** You now know whether the platform you are looking at will give you one, and what to do on the day if it will not.

**You are a vendor whose competitors are cited from review sites.** You know which parts of a profile are open to reading, so you know which part of your presence there can be quoted.

**You are writing a research post and you planned to quote review data.** You have a dated, reproducible measurement of what is available, and a clean way to describe the gap without guessing.

**You are building a crawler and you were told this category is closed.** It is not closed in the way the folklore says. It is open, selectively, on paths that the rules permit, and it is closed at the door in a way that will not move.

What this study does not tell you

Where this census sits in the rest of our work

Three studies that bear on the same question: what a vendor's own page lets you find out before you talk to them.

How we checked

We probed 14 review platforms on 23 October 2026. For each host we made one request for the rules file as an unconfigured client, one as a browser, and we repeated the unconfigured request hours later to check that the answer was repeatable. Where the apex host did not resolve, we tried the www host as well. Every response was classified before any rule was read: a response counted as a rules file only if it carried User-agent lines and no HTML, so that a challenge page served with a 200 could not be mistaken for permission.

On the hosts that served a real rules file, we read the group that applies to all crawlers and applied it to three representative product profile paths per host, using the longest-match precedence the major crawlers use. We then classified each rule by what its path pattern targets, separating rules that block reading a page from rules that block acting on one. Every figure on this page is a field of the recorded probe data, and the rules files themselves are stored with their hashes so the readings can be re-checked.

We fetched no product profile, no review and no comparison page from any host in this study. The request under a browser identity was used to read the rules file itself, because the point of the study is to find out whether a policy exists, and a policy that cannot be obtained is itself a finding.