The file must contain a link to the sitemap

[subornaakter20]

Robots.txt

The robots.txt file must be on each resource. It is located at sitename.ru/robots.txt. It is this document that search robots access first: there, developers leave recommendations on what needs to be indexed and shown to users, and what not.

Recommendations are made individually for each site, but there are several general rules:

It is necessary to specify the User-agent for us companies board of directors email list each search engine. In the case of Yandex, this is YandexBot.

Service pages such as the admin panel, registration form and shopping cart should be closed from indexing.


Site map. Sitemap.xml file

The sitemap shows bots the structure of the resource with all its pages. This allows you to speed up the indexing process.

The map should not contain service pages, registration and authorization forms, etc. More detailed information about the rules for its compilation can be found in Yandex.Help.

Main mirror of the site

Yandex recently announced that it would gradually phase out the Host directive. Instead, it will use 301 redirects.

Don't forget to select the main site mirror — version with or without www. Robots should receive a "200 OK" response from this mirror. To check this, use Yandex.Webmaster tools. Avoid using intermediate redirects: they make pages process more slowly.

Website optimization for mobile devices

Yandex has a special algorithm Vladivostok, which was created to evaluate the optimization of a site for mobile devices as one of the ranking factors.

High quality of the site implies its correct display on smartphones and tablets. At the moment, this criterion is fundamental, which is not surprising, because in some industries mobile traffic exceeds computer traffic.

You can check the correct operation of the mobile version of the site through Yandex.Webmaster.

Turbo pages

Technical website optimization

In 2017, Yandex introduced a feature for creating turbo pages. You may have already seen similar ones on your smartphones: they are indicated by an icon in the form of a rocket.

Source: shutterstock.com

Turbo pages load 15 times faster, and they rank higher in mobile search results than regular pages.

Today, this feature is most relevant for information sites and media. It is expected that in the future, support for online stores will appear and turbo pages will be able to create commercial resources, among other things.