To block crawler access, “.htaccess” file needs to be edited.

To block “bingbot” and “MSNbot” crawlers as an example, edit “.htaccess” file, and at the end of file add the following code:

RewriteCond %{HTTP_USER_AGENT} 'bingbot|MSNbot'    
RewriteRule . * - [F,L]

Most common crawlers include: Googlebot, bingbot, MSNbot, YandexBot, SemrushBot-SA, SemrushBot, etc.