Crawler Set-Up
Allowing Powerbeans Crawler Access
To ensure the Powerbeans crawler can extract article content from your site correctly, please review your CDN and firewall configurations.
If You Use Cloudflare
1.1) Create a Firewall Rule
Log in to the Cloudflare Dashboard.
Go to Security → WAF (Firewall) → Firewall Rules.
Click Create a Firewall Rule.
1.2) Define the Rule Conditions
Field: http.user_agent
Operator: contains
Value:
PowerBeansScraper/1.0
(Optional) Use equals for exact matching if preferred.
1.3) Set the Action
Action: Allow
This ensures requests from our crawler bypass security filters such as Bot Fight Mode, JS challenges, and country blocks.
1.4) (Optional) Add a Descriptive Name
Example: Allow Powerbeans Scraper
1.1) Create a Firewall Rule
Log in to the Cloudflare Dashboard.
Go to Security → WAF (Firewall) → Firewall Rules.
Click Create a Firewall Rule.
1.2) Define the Rule Conditions
Field: http.user_agent
Operator: contains
Value:
PowerBeansScraper/1.0
(Optional) Use equals for exact matching if preferred.
1.3) Set the Action
Action: Allow
This ensures requests from our crawler bypass security filters such as Bot Fight Mode, JS challenges, and country blocks.
1.4) (Optional) Add a Descriptive Name
Example: Allow Powerbeans Scraper
1.1) Create a Firewall Rule
Log in to the Cloudflare Dashboard.
Go to Security → WAF (Firewall) → Firewall Rules.
Click Create a Firewall Rule.
1.2) Define the Rule Conditions
Field: http.user_agent
Operator: contains
Value:
PowerBeansScraper/1.0
(Optional) Use equals for exact matching if preferred.
1.3) Set the Action
Action: Allow
This ensures requests from our crawler bypass security filters such as Bot Fight Mode, JS challenges, and country blocks.
1.4) (Optional) Add a Descriptive Name
Example: Allow Powerbeans Scraper
If You Use Another CDN or Firewall (e.g., Akamai, Fastly, AWS WAF, Imperva, etc.)
Allow or whitelist HTTP requests where the User-Agent header contains PowerBeansScraper/1.0.
Ensure such requests are not blocked or rate-limited.
Verify that article URLs can be fetched successfully (HTTP 200) from your public endpoints when using this User-Agent.
Allow or whitelist HTTP requests where the User-Agent header contains PowerBeansScraper/1.0.
Ensure such requests are not blocked or rate-limited.
Verify that article URLs can be fetched successfully (HTTP 200) from your public endpoints when using this User-Agent.
Allow or whitelist HTTP requests where the User-Agent header contains PowerBeansScraper/1.0.
Ensure such requests are not blocked or rate-limited.
Verify that article URLs can be fetched successfully (HTTP 200) from your public endpoints when using this User-Agent.
Last edited: 13/11/2025
Last edited: 13/11/2025