Search Engines use crawlers to crawl the websites, like Googlebot for google, Bingbot for bing. When users peform searches, the search engines perform a filter and check within its index, and deliver results. If search engines can't find the page due to bad architecture, your site won't appear in search results.
Major factors which blocks search engines to access sites are bad internal linking, slow page loads, URL errors, noindex signals. Many sites don't face these seroius problems, but you should be aware of the above key factors that prevents search engines from accessing site.
Making use of Robots.txt and telling google what to crawl and what not to, is a ideal and most preferred method. Make effective usage of crawl budget. Make bots discover all your indexable pages.
Mobile friendliness is not just about your site looks on a mobile or tab, but it also about how your site works on these devices. In March 2018, Google started implementing mobile-first indexing and now it completely rolled out. Many CMS support mobile first indexing, but make sure the navigation and performace over mobile devices is as good as desktop.
Leverage Google's Accelerated Mobile Pages to deliver your content super fast. Common mistakes in mobile friendliness are slow page loads, icons too close to use, overlap of buttons, navigation difficulties. Many of these factors decreases user experience on site.
Duplicate content is always bad for search engine rankings, having duplicate content everywhere makes search engines difficult to identify and bring results when a search is performed. When Google finds similar content from many sources, they consolidate into one and helps users with the best page.
Usage of canonical tags are highly required to signal google about our preferred page. Effective pagination, navigation always allows bots and users the best user experience. Always optimise your pages for users and eventually helps bots for easy accessing and understanding.
Website speed is one of the major ranking factors of google, your website should load fast whether the user opens it on desktop or mobile. Sites loading faster have the advantage of SEO. Fast loading times reduces the bounce rate and improves user engagement ,also user experience is required.
Check your page speed with Google's Page Speed insights tool, Example on how we optimised our page speed to perform best Data As Services Page Speed
Google always suggests websites to be on HTTPS. Google prefers HTTPS sites than HTTP for rankings. Assets should also be through HTTPS protocols. 68% of total websites are now on HTTPS.
HTTPS is a great addition to SEO value of your site, as it tells users that your site is secure. Security levels also a ranking factor by google.
URLs to have descriptive text, as they give first glance about the page in search results, which impact positive results in seo, better user engagement. Always create URL structure that are human understandable and easy to access.
URLs with ids, malformed urls tend to decrease their organic rankings slowly in the search. Clean URLs always leave a positive and better organic performance. Make usage of Directories to group similar pages into categories, helps users and bots to understand better about the site.