Unable to map route for robots.txt in asp.net mvc

Question

I am developing an asp.net mvc application. I am creating robots.txt for my application to prevent from bots because my current site is getting many robot requests. So I found this link, Robots.txt file in MVC.NET 4 to create robots.txt. But I when I access my application like this entering url, "www.domain.com/robots.txt", it is always returning 404 page.

This is my action method in HomeController

public ActionResult Robots()
{
    Response.ContentType = "text/plain";
    return View();
}

This is my robots view

@{
    Layout = null;
}

User-agent:*
Disallow:/

I configured route for robots.txt like this in RouteConfig

 public static void RegisterRoutes(RouteCollection routes)
 {
            routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

            //routes.MapMvcAttributeRoutes();
            routes.MapRoute(
               "Robots.txt",
               "robots.txt",
               new { controller = "Home", action = "Robots" },
               new string[] { "AyarDirectory.Web.Controllers" }
               );

          //other routes
}

But when I access this url, "www.domain.com/robots.txt", it is always returning 404 page. How can I add robots.txt correctly to my application?

The file extension `.txt` in the route is what is causing your problem. — Nkosi, Aug 07 '16 at 06:02
But if i map like this "robots", it will not filter robots. Robots always look for /robots.txt. Is it? If I exclude .txt, will it still filter robots please? — Wai Yan Hein, Aug 07 '16 at 06:06
no it wont. you need to configure to allow robot.txt take a look at this and see if it helps http://stackoverflow.com/a/36151756/5233410 — Nkosi, Aug 07 '16 at 06:07

score 10 · Accepted Answer · answered Aug 07 '16 at 06:13

Creating a route ending with a file extension is not allowed by default in ASP.NET MVC. To get around this security restriction, you need to add the following to the Web.config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!-- ...Omitted -->
  <system.webServer>
    <!-- ...Omitted -->
    <handlers>
      <!-- ...Omitted -->
      <add name="RobotsText"
           path="robots.txt"
           verb="GET"
           type="System.Web.Handlers.TransferRequestHandler"
           preCondition="integratedMode,runtimeVersionv4.0" />
    </handlers>
  </system.webServer>
</configuration>

score 7 · Answer 2 · answered Jul 11 '19 at 10:00

7

In Asp.net Core you can simply add your robots.txt file to the wwwroot directory.

answered Jul 11 '19 at 10:00

Johan Maes

1,161
13
13

score 1 · Answer 3 · answered Jan 17 '18 at 17:20

Would say same thing as above you need to change the web.config to allow the route with .txt to work. I had same issue with a project I was working on and I got it to work.

However if you using a view for the output of the robots without a model you might as well keep a static robots.txt as it will give you no advantage. Another way is to output the text direct from the action using a string builder.

score 0 · Answer 4 · answered Nov 22 '21 at 02:13

Nkosi's System.Web.Handlers.TransferRequestHandler approach in web.config approach did not work for me due to the environment I was working in, resulting in 500 errors.

An alternative of using a web.config url rewrite rule worked for me instead:

<rewrite>
    <rules>
        <rule name="Dynamic robots.txt" stopProcessing="true">
            <match url="robots.txt" />
            <action type="Rewrite" url="/DynamicFiles/RobotsTxt" />
        </rule>
    </rules>
</rewrite>

Unable to map route for robots.txt in asp.net mvc

4 Answers4