|
|
# `SERPent` |
|
|
|
|
|
## SERP results scrapping |
|
|
|
|
|
SERPent exposes an unified API to query SERP (Search Engine Result Pages) for a few common search engines, namely: |
|
|
|
|
|
- DuckDuckGo |
|
|
- Brave |
|
|
- Bing |
|
|
- Google Patents |
|
|
- arXiv |
|
|
- Google |
|
|
|
|
|
The application uses the `playwright` library to control a headless web browser, to simulate normal user activity, to fool the anti-bot measures often present on those sites. See the `/serp/` endpoints for search results scrapping. |
|
|
|
|
|
|
|
|
## Website sources scrapping |
|
|
|
|
|
SERPent also exposes a few endpoints to scrap the contents of certain sources (patents, scholar). See the `/scrap/` endpoints for supported website sources scrapping. |
|
|
|
|
|
|
|
|
|
|
|
|
