--- title: "Product Review Data for Sentiment Analysis: What to Collect" description: "Which review fields to keep from Trustpilot, Google Play and the App Store, how to handle rating skew and duplicates, and why to drop reviewer names." url: https://perceptlayer.com/blog/product-review-data-sentiment-analysis --- [Blog](https://perceptlayer.com/blog) / [Web data by industry](https://perceptlayer.com/blog/category/web-data) # Product Review Data for Sentiment Analysis: What to Collect 15 Sept 2026 Perceptlayer 11 min read For sentiment analysis, collect four things per review: the text, the star rating, the publish date and a stable review ID. Add a language tag, a verification flag and a helpful-vote count when the source has them. Drop reviewer names, avatars and usernames before anything is stored. This article compares what Trustpilot, Google Play and the Apple App Store return, from recorded responses. ## Key takeaways - Keep `text`, `rating`, a date and the source's review `id` per review; add language where the source returns it. Everything else is optional context. - Reviewer names sit in different fields per source ( `consumer.displayName`, `author`, and the Google Play review `title`). Drop them at ingestion. - Rating skew is real: the recorded Trustpilot sample for one retailer has 33,636 one-star ratings out of 49,830 (67.5%). One retailer's profile cannot be set against an app's score on another platform: the product changes along with the platform. - Deduplicate on `(source, id)` first, then on a hash of normalised text, skipping short texts. Paginated pulls overlap when new reviews arrive mid-run. - Each call costs 1 credit. If a call returned 20 reviews, 10,000 reviews would be 500 calls × 1 credit = 500 credits; check the page size of what you actually get back. ## Which review fields does a sentiment model need? A sentiment model needs the review text, a label or weak label (the star rating), a timestamp so you can chart change, and an ID so you can deduplicate and re-run. Language matters where you have it, because a classifier built for one language will misread reviews written in another, and the Trustpilot sample mixes German and English. Only Trustpilot returns a language per review in the recorded samples, so for the other two you will need a language detector. Everything else, such as images, profile links and reviewer history, adds privacy risk without improving the score. Here is how the three sources line up in the recorded responses. "Not in sample" means the field isn't in the recorded excerpt, not that it can't exist. | Need | Trustpilot ( `company-reviews`) | Google Play ( `product-reviews`) | App Store ( `product-reviews`) | | --- | --- | --- | --- | | Review ID | `reviews[].id` | `reviews[].id` | `reviews[].id` | | Rating | `rating` | `rating` | `rating` | | Text | `text` | `snippet` (unconfirmed whether full body) | `text` | | Headline | `title` | Not in sample ( `title` is the reviewer's name) | `title` | | Date | `dates.publishedDate`, `dates.experiencedDate` | `iso_date`, `date` | `date` | | Language | `language` | Not in sample ( `hl` is a request parameter) | Not in sample | | Helpful votes | `likes` | `likes` | Not in sample | | Verification | `isVerified`, `labels.verification` | Not in sample | Not in sample | | Edited flag | Not in sample | Not in sample | `is_edited` | | Reviewer identity | `consumer.*` | `title`, `avatar` | `author` | | Star histogram | `filters.reviewStatistics.ratings` | `ratings[]` | Not in sample | Two details stand out. Google Play puts the reviewer's name in a field called `title`, so a pipeline that treats `title` as a headline will index names as review text. And neither the Google Play nor the App Store excerpt has a per-review language field, so run a language detector on `text` for those two. ## What does each source return per review? Each source returns a flat list of reviews plus context about the product, and the context is often useful. Trustpilot adds a business profile and rating counts, Google Play adds app metadata and a per-star histogram, and the App Store response in the recorded sample is reviews and pagination only. Request shape differs slightly per source. ### Trustpilot [`GET /v1/trustpilot/company-reviews`](https://perceptlayer.com/apis/trustpilot/company-reviews) takes a required `company_domain` and filters such as `rating`, `verified`, `with_replies`, `date_posted`, `languages` and `sort` ( `relevance` or `recency`). ```bash curl "https://api.perceptlayer.com/v1/trustpilot/company-reviews?company_domain=amazon.com&languages=all" \ -H "X-API-Key: $API_KEY" ``` A review in the recorded response looks like this, trimmed: ```json { "id": "6adaa27868a81eadd98e9c53", "text": "Disgusting company run by disgusting and greedy people ", "rating": 1, "title": "Disgusting company run by disgusting…", "likes": 0, "source": "Organic", "dates": { "experiencedDate": "2026-10-22T00:00:00.000Z", "publishedDate": "2026-10-23T01:55:36.000Z" }, "language": "en", "isVerified": false } ``` The same object also carries a `consumer` block with a display name, image URL and country code. Keep `consumer.countryCode` if you need a coarse region; drop the rest. ### Google Play The [Google Play API](https://perceptlayer.com/apis/google-play) exposes reviews through [`GET /v1/google-play/product-reviews`](https://perceptlayer.com/apis/google-play/product-reviews), which takes an app ID in `product_id` (or `id`, or a `url`) and optional `rating`, `sort_by` ( `most_relevant`, `newest`, `rating`), `platform`, `hl`, `gl`, `num` and `next_page_token`. ```bash curl "https://api.perceptlayer.com/v1/google-play/product-reviews?product_id=com.spotify.music&store=apps&num=20" \ -H "X-API-Key: $API_KEY" ``` ```json "reviews": [ { "id": "6b8cd845-6fcd-4296-b05e-9e1b3feada62", "rating": 1, "snippet": "Spotify has a new, horrendous feature where the ending to the song that you are listening to gets cut off by the middle of the next song. I have had spotify for…", "likes": 491, "date": "September 15, 2026", "iso_date": "2026-09-15T22:20:33Z" } ] ``` I removed the `title` and `avatar` fields from that excerpt on purpose: they identify the reviewer. Check `snippet` before you score it. Both recorded snippets end in `…`, so it is unconfirmed whether `snippet` holds the full review body or a shortened preview. The recorded excerpts are trimmed for display, which makes the ending alone inconclusive. Before scoring, log the character length of each `snippet` and look at the distribution: if many values stop at a similar length or end in `…`, treat the field as truncated, don't score it as the whole review, and say so in your results. The Python script below counts these. The response also includes `product_info.rating` (4.3 in the sample) and a `ratings` array of `stars` and `count` pairs. ### Apple App Store [`GET /v1/apple-app-store/product-reviews`](https://perceptlayer.com/apis/apple-app-store) takes `product_id` and `store`, plus `country`, `num`, `offset` and `next_page_token`. There is no `rating` or sort parameter listed, so filter by rating after you collect. ```bash curl "https://api.perceptlayer.com/v1/apple-app-store/product-reviews?product_id=324684580&store=apps&num=20" \ -H "X-API-Key: $API_KEY" ``` ```json { "id": "9447841001", "title": "Stop changing the queue!!!!!", "rating": 5, "text": "Stop changing the way the queue works every single time I use this app! I swear Im this close to subscribing to apple music …", "date": "2022-12-29T14:48:34Z", "is_edited": false } ``` Note the dates. The two reviews in the recorded excerpt are from 2022 and 2024, while the Google Play sample's are from September 2026. Check the date range of what you pull before you treat a sample as "current" sentiment. For a fuller look at one source, see the [App Store reviews article](https://perceptlayer.com/blog/app-store-reviews-api), the [Google Play article](https://perceptlayer.com/blog/google-play-store-api) and the [Trustpilot article](https://perceptlayer.com/blog/trustpilot-reviews-api). ## How should you handle rating skew? Treat the star distribution as a property of the source, not of the product. In the recorded Trustpilot response for one large retailer, `ratings` shows 33,636 one-star, 3,004 two-star, 1,869 three-star, 2,423 four-star and 8,898 five-star, out of 49,830. That is 67.5% one-star and 17.9% five-star, with little in between, and the profile's `trustScore` is 1.5. The Trustpilot sample is lopsided, and I can't tell from one response why. Its recorded reviews carry a `source` of `Organic`, a `verificationSource` of `invitation`, a `verificationLevel` of `not-verified` and `isVerified` set to `false`. I'm reporting those values as recorded; the response doesn't document what they mean together, so test them against your own data rather than assuming. The Spotify sample on Google Play reads 4.3 overall ( `product_info.rating`). Setting that against the retailer's 1.5 shows nothing about platforms, because the samples differ in both product and platform and the two effects can't be separated from these responses. To compare platforms you would need the same product on both. Practical steps, tied to the fields above: 1. **Store `rating` next to `text`** and report sentiment per star level, not one blended score. 2. **Keep `verified` and `source` where they exist.** Trustpilot's `company-reviews` accepts a `verified` filter, and each review carries `isVerified` and `source`, so you can check whether your result changes when you restrict to verified reviews. 3. **Sample a fixed number of reviews per star, then reweight.** A plain newest-first pull on a skewed profile gives you mostly one-star text and very little else to learn from. The `rating` filter on Trustpilot and Google Play lets you pull each star level separately. ### A fixed-per-star sample The method: pull the same number of reviews for each of the five star levels, score each level, then weight the per-star results by the source's own star histogram so the blended number still reflects the real mix. 1. **Pull N reviews per star.** Call `company-reviews` (Trustpilot) or `product-reviews` (Google Play) once per star with `rating` set, and page until you have N. Google Play's `rating` accepts `1` to `5`. Trustpilot's `rating` is documented only as a string with no listed values, so test `rating=1` first and check that `filters.totalNumberOfFilteredReviews` in the response drops below the unfiltered total before you rely on it. 2. **Read the true mix from the histogram.** Trustpilot returns it in `filters.reviewStatistics.ratings` ( `one` to `five`); Google Play returns `ratings[]` as `stars` and `count` pairs. In the recorded Trustpilot response that is 33,636 of 49,830 for one star, which gives the weights. 3. **Score each star's sample separately**, then compute the weighted average: each star's mean score times that star's share of the histogram. The App Store has no `rating` filter in its parameter list, so for that source collect the pages you need and split by `rating` afterwards. Cost is calls × 1 credit. Assume 20 reviews per call (the recorded Trustpilot response reports `perPage` as 20; the Google Play request asked for `num=20`, and page sizes aren't documented, so measure yours). Then N = 100 reviews per star is 5 calls per star, 25 calls for five stars, so 25 credits per source. Doing it on Trustpilot and Google Play is 50 credits. If a star level has fewer than N reviews, you get fewer and the cost drops. Keep in mind that a 100-review sample per star is small, so treat the per-star scores as rough and raise N if the weighted result changes a decision. ## How do you deduplicate reviews? Deduplicate in two passes: exact on the source's own review ID, then fuzzy on normalised text. The first catches overlapping pages and repeated runs. The second catches the same complaint pasted across accounts or platforms, which an ID can't see because IDs are only unique within one source. Overlap happens for ordinary reasons. If you page through a feed that's receiving new reviews, every new review pushes older ones down a page, so page two repeats the end of page one. Trustpilot's response also returns a `relevantReviews` list beside `reviews`. In the recorded sample the IDs in the two lists don't overlap, but they are separate lists and may overlap in other pulls, so I'd keep `relevantReviews` out of a time series. To check, intersect the `id` sets of the two lists before you merge them. Use a composite key of source plus ID. Then lowercase the text, strip whitespace and punctuation, and hash it to flag identical bodies. Sort by date first so the copy you keep is the earliest one, and only text-hash above a minimum length: short texts such as "Great app" collapse to one hash and would drop real, separate reviews. Record how many duplicates you dropped rather than deleting silently; the script below counts them. The App Store's `is_edited` field is a boolean, `false` on both recorded reviews. Store it with each record so you can see which reviews are marked as edited. ## Why should you drop reviewer names? Drop them because sentiment analysis doesn't need them, and keeping them turns a text dataset into a personal-data dataset. Under the GDPR, [personal data](https://gdpr-info.eu/art-4-gdpr/) (Article 4(1), Regulation (EU) 2016/679, 2016) is any information relating to an identified or identifiable natural person, and the definition lists identifiers such as a name, location data or an online identifier. The UK regulator's [guidance on personal data](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/personal-information-what-is-it/what-is-personal-data/what-is-personal-data/) (ICO, checked 2026) quotes the same wording for the UK GDPR. So a display name attached to a country code and a profile image can fall within it. Whether a given record does depends on your context. This isn't legal advice; check your own obligations. The identifying fields in the three recorded responses are: | Source | Fields to drop | | --- | --- | | Trustpilot | `consumer.displayName`, `consumer.id`, `consumer.imageUrl`, `consumer.numberOfReviews` | | Google Play | `title` (the name), `avatar` | | App Store | `author` | Drop them in the ingestion function, before the write, so they never reach a table, a log or a model prompt. Reviews can also contain names in the body text, so if your use case allows it, run a pass over `text` to mask obvious personal details. Collected data here comes from publicly available sources and is returned as found at the time of the call, which doesn't remove your responsibility for how you store it. ## How do you collect and normalise all three in Python? Write one small function per source that returns the same flat record, with identity fields left out. Then one loop deduplicates and counts ratings. The script below follows the response shapes recorded above and runs with your key set in `API_KEY`. ```python import hashlib import os import re from collections import Counter import requests BASE = "https://api.perceptlayer.com" HEADERS = {"X-API-Key": os.environ["API_KEY"]} def get(path, **params): r = requests.get(BASE + path, headers=HEADERS, params=params, timeout=30) r.raise_for_status() return r.json() def trustpilot(domain, pages=2): for page in range(1, pages + 1): data = get("/v1/trustpilot/company-reviews", company_domain=domain, languages="all", page=page) for r in data.get("reviews", []): yield {"source": "trustpilot", "id": r["id"], "rating": r["rating"], "text": r.get("text", ""), "date": r["dates"]["publishedDate"], "lang": r.get("language")} def google_play(app_id, pages=2): token = None for _ in range(pages): params = {"product_id": app_id, "store": "apps", "num": 20} if token: params["next_page_token"] = token data = get("/v1/google-play/product-reviews", **params) for r in data.get("reviews", []): yield {"source": "google-play", "id": r["id"], "rating": r["rating"], "text": r.get("snippet", ""), "date": r["iso_date"], "lang": None} token = data.get("pagination", {}).get("next_page_token") if not token: break def app_store(app_id, pages=2): token = None for _ in range(pages): params = {"product_id": app_id, "store": "apps", "num": 20} if token: params["next_page_token"] = token data = get("/v1/apple-app-store/product-reviews", **params) for r in data.get("reviews", []): yield {"source": "app-store", "id": r["id"], "rating": r["rating"], "text": r.get("text", ""), "date": r["date"], "lang": None} page = data.get("reviews_pagination", {}) token = page.get("next_page_token") if page.get("end_of_history") or not token: break def text_hash(text): norm = re.sub(r"\W+", " ", text.lower()).strip() return hashlib.sha1(norm.encode()).hexdigest() MIN_HASH_CHARS = 20 # shorter texts ("Great app") are not text-hashed records = sorted([*trustpilot("amazon.com"), *google_play("com.spotify.music"), *app_store("324684580")], key=lambda rec: rec["date"]) seen_ids, seen_text, rows, dupes = set(), set(), [], Counter() for rec in records: key = (rec["source"], rec["id"]) h = text_hash(rec["text"]) if len(rec["text"]) >= MIN_HASH_CHARS else None if key in seen_ids or (h and h in seen_text): dupes[rec["source"]] += 1 continue seen_ids.add(key) if h: seen_text.add(h) rows.append(rec) by_source = Counter((r["source"], r["rating"]) for r in rows) for (source, stars), n in sorted(by_source.items()): print(source, stars, n) print("duplicates dropped:", dict(dupes)) # Google Play `snippet` may be a preview, not the full body: check before scoring. gp = [r["text"] for r in rows if r["source"] == "google-play"] cut = sum(t.endswith("…") for t in gp) print("google-play texts ending in an ellipsis:", cut, "of", len(gp)) print("google-play text lengths:", sorted(len(t) for t in gp)) # Only Trustpilot returns a language; the rest go to a detector of your choice. langs = Counter(r["lang"] or "unknown" for r in rows) print("languages:", dict(langs)) ``` Sorting by `date` first means the earliest copy is the one kept. The records contain no names, avatars or usernames because the functions never read them. A failed call returns an HTTP status and a JSON error body, and `raise_for_status()` surfaces it; see [the errors reference](https://docs.perceptlayer.com/errors) for the codes. Failed calls aren't billed. ### What does it cost? Each call is one credit, whichever source you use. As a worked example, assume a call returns 20 reviews (page sizes aren't documented, so measure yours). Then 10,000 reviews from one source is 500 calls × 1 credit = 500 credits, and all three sources at that depth is 1,500 credits. The free tier gives 1,000 credits a month, which covers two of the three sources; the remaining 500 credits need a pack. See [credits](https://docs.perceptlayer.com/credits) for how billing works and [pricing](https://perceptlayer.com/pricing) for what credits cost. ## When is this not the right data? If you only need your own app's reviews, the store owners' own tools are worth checking first, since they are built around your account. Perceptlayer returns what is publicly visible, as found at the time of the call. For a market-wide view across competitors, or for a source without an official API, a single key across sources saves you maintaining three integrations. Also consider volume. Sentiment on a few thousand recent reviews often answers the question; a full historical backfill for a product with millions of reviews is a large number of calls. Decide the date window first, then size the job. For other sources of review data, see the [Reviews APIs](https://perceptlayer.com/verticals/reviews) and [Apps and media APIs](https://perceptlayer.com/verticals/apps) hubs, and the related [hotel price data](https://perceptlayer.com/blog/hotel-price-data-rate-monitoring) article for a similar fields-first approach. To try it, [get a free API key](https://perceptlayer.com/signup), open the playground on the [Trustpilot API page](https://perceptlayer.com/apis/trustpilot), and run the `company-reviews` call for a domain you know. ## APIs in this post - [Trustpilot **1 credit per call** · 6 endpoints](https://perceptlayer.com/apis/trustpilot) - [Google Play **1 credit per call** · 9 endpoints](https://perceptlayer.com/apis/google-play) - [Apple App Store **1 credit per call** · 13 endpoints](https://perceptlayer.com/apis/apple-app-store) ## Frequently asked questions ### Which fields are the minimum for sentiment analysis? The text, the rating, a date and the source's review ID, plus a language tag where the source returns one. The text is what you score, the rating gives a label to check against, the date lets you chart trends, the ID prevents double counting, and the language picks the right model. Everything else is optional context, and identity fields should be dropped. ### Can I compare average ratings between Trustpilot and Google Play? Not from these samples. The recorded Trustpilot response is for one retailer ( `trustScore` 1.5, 67.5% one-star ratings) and the Google Play one is for the Spotify app (4.3), so product and platform differ at once and can't be separated. Compare the same product on both, or compare a source against its own history. ### How many credits does it cost to pull reviews? One credit per successful call on each of these endpoints, and failed calls cost nothing. If a call returns 20 reviews (an assumption; check your own responses), 5,000 reviews is 250 calls × 1 credit = 250 credits. Free accounts get 1,000 credits a month after verifying an email. Check [pricing](https://perceptlayer.com/pricing) for what larger packs cost. ### Do I need the restricted-endpoint terms to use review data? The endpoints in this article are standard tier, so no. Restricted endpoints cover social, personal and SEO data and need the terms accepted in the dashboard. Separately, reviews may contain personal data, so decide your lawful basis and drop identity fields at ingestion, as covered above. ### What happens if I hit the rate limit during a big pull? You get a 429 response with a `Retry-After` header giving the seconds to wait. Limits are per account and counted per minute; free accounts get 1 request a second. Sleep for the header value and continue. Because requests have no idempotency key, a timed-out request may have been billed, so dedupe by ID. See [rate limits](https://docs.perceptlayer.com/rate-limits). sentiment analysis product reviews trustpilot google play app store ## More from the blog 1. [Web data by industry](https://perceptlayer.com/blog/category/web-data) 21 Oct 2026 Perceptlayer 8 min read ## [Job Posting Data for Labour-Market Research](https://perceptlayer.com/blog/job-posting-data-labour-market-research) Which Indeed posting fields support trend analysis, where postings mislead versus hires, and a weekly collection plan that costs 3,000 credits a month. 2. [Web data by industry](https://perceptlayer.com/blog/category/web-data) 6 Oct 2026 Perceptlayer 7 min read ## [Salary Data from Job Sites: Reading Ranges Without Bias](https://perceptlayer.com/blog/salary-data-job-sites-bias) Glassdoor's salaries endpoint returns p10 to p90 total pay, a currency and a national count. How to read those ranges and the biases of self-reported pay. 3. [Web data by industry](https://perceptlayer.com/blog/category/web-data) 30 Sept 2026 Perceptlayer 7 min read ## [Software Review Data Analysis for Competitive Research](https://perceptlayer.com/blog/software-review-data-competitive-research) What Capterra, Trustpilot and Google Play reviews can and can't tell you about a competitor, which fields to keep, the sampling bias, and a worked rating tally.