How SEO Experts Use Screaming Frog? Pro Tips

Advanced Screaming Frog crawling configurations, custom extraction regexes, and API integrations for technical audits.

Speakers: Samet Özsüleyman, Aybüke Selim

16 Apr 2024

Video Transcript

⏱ 25 min read• 4,434 words

Read the full speech or switch languages

— Hi everyone. Welcome to Zeo Talks. Today, Aybüke and I are discussing Scrum FROG. It’s easily the top tool for SEO pros. Aybüke, welcome back.

— Thanks, Samet.

— How are you?

— I didn't want us to wait too long.

— Exactly. Good call.

— Right after our podcast.

— I'm doing great. Everything is going well.

— Awesome. Like you mentioned, we'll talk about Scrum FROG. Scrum FROG is an endless tool. It gets constant updates. New features might drop after we record this. This isn't just a basic Scrum FROG tutorial. We'll focus on what SEO experts prioritize. We'll cover reporting and critical data points. We'll show how to guide brands and brief developer teams. We’ll share our personal recommendations. Let’s start with the modes in Scrum FROG. Can you explain the modes in Scrum FROG? Like spider and list mode?

How do we use them? When are they most useful?

— In spider mode, you enter a URL to crawl the whole site. List mode lets you crawl specific URLs. You can upload a list of URLs. Say you have 300 URLs. You can just crawl and analyze that specific list. We can feed this URL list in. Spider mode has tons of features. We'll cover those soon. You don't have to crawl every URL on the domain. Use include and exclude settings. You can crawl specific URL parameters. Spider mode handles full site crawls easily. There is also a sitemap section. You can run sitemap crawls in list mode.

It used to be limited to one sitemap. You added one sitemap URL and it crawled that. Now you can crawl multiple sitemap URL'leri at once. You can analyze just those specific URLs. Let's talk about compare mode. You can compare two different crawls. Say you ran a crawl last month. You sent the technical fixes to your team. The team made site improvements. Now you want to see the current status. Use compare mode to see the progress. Did 404 errors drop? It makes comparisons super simple.

Did I miss anything? Let me check.

— That happens. Scrum FROG has so many details. I can jump in here. We use these modes in SEO projects all the time. Take an audit project, for example. Before we start, we check the static code ratio. We track how we improve those numbers. Or we track totally different KPIs. As you mentioned, it’s great for before-and-after analysis. Say we have a sitemap for a brand like Nike. We might suspect issues with that sitemap. Maybe there are technical glitches or other problems. For instance, my brand sitemap URLs might have stockout issues. I’m just giving examples. We use Scrum FROG to pinpoint these issues exactly.

Say we suggest changing 200 page titles or meta tags. Once the brand team implements them, they ask us to check. We just open list mode. We drop those 200 URLs in. We don't need to crawl the whole site. We save time and spot errors with surgical precision.

— Exactly. Great additions.

— I wanted to ask you to expand on segmentation. You covered spider mode and list mode. You went over the other modes too. But segmentation is a newer feature in the Scrum FROG tool. How do we use segmentation in our projects? Can you explain that?

— I was just about to jump into that. Perfect timing. Think of segmentation as a module. It simplifies tracking and analysis. It’s a huge help for large sites. Segmentation lets us group site pages by custom rules. Here is an example. Say you have an e-commerce site. You have product pages and blog pages. You might have many different categories. We set rules based on URL paths. This lets us segment data. We can see what percentage of blog pages return 200. We can also spot missing meta titles in specific segments.

Remember the compare mode we mentioned? You can use segmentation rules there too. It lets you compare changes across category pages. It really gives us a massive advantage. It makes auditing sites much faster. You might have 2,000 URLs. Analyzing them all at once isn't always efficient. This helps us spot errors quickly. We can take action much faster. Like I said, it’s useful for the blog side. We got a warning about duplicate content. Duplicate pages are showing up. Segmentation makes tracking these issues much easier.

Anything you want to add?

— Yes. Let me show you. Prioritization is key when you have a massive site. Segmentation helps a lot. As you mentioned, blogs can be quite problematic. Different teams often handle blogs and product pages. We can isolate blog errors and send them to the right team. It’s not an issue for small sites. You might say, "I'm only crawling 2,000 pages, I don't need segmentation." You’re right for your case. But we work with huge sites. Segmentation is invaluable for us. Let me show you. I’m opening Scrum FROG now. You can see the data for the Zeo website here.

I’ve already set up a rule. Then we go to segmentation. We can see all our segments and even color-code them. If you want to segment blog pages. You can label them as "Resources" or "Blog" depending on your site. If a URL contains "blog," label it as such. Define these segments directly. You might have blogs, product pages, or other types. Group everything else as "other." Proper segmentation speeds up your analysis. It helps you spot problems faster. Always set up these segments when starting a project.

If the site size allows, save these rules locally. It’s great for before-and-after comparisons. I wanted to show you this segmentation, Aybüke.

— Awesome. Seeing it makes everything clearer. It’s all clicking now.

— Definitely.

— Great job. I can show what I mentioned earlier. Let’s look at the initial Scrum FROG screen. I haven't run a crawl yet. This is the screen you see when you first open Scrum FROG. You can use the free version too. We use the licensed one. But Scrum FROG has a 500 URL limit. It’s enough to test the features. You can see what it can do. Spider mode is on by default. You can see other modes here. Right below spider mode is list mode. If you have a URL list, just upload it here.

Add your list manually. You can also analyze sitemap URLs. Just paste your sitemap URL here. There is also a section for Google Sheet. You can add your Google Sheet URL. Just grant access and add the Sheet URL. I just noticed the SERP mode. You add your URL, title, and description to a Google Sheet. It shows how it looks in the SERP. You can check pixel counts. See if your title tag is too long. You can see title tag errors directly here. There are other online tools for this. They are good for quick actions. But use SERP mode if you have a huge list.

I mentioned compare mode before. Let me show it again. Upload two files and click compare. It makes the comparison clear. Let’s look at the existing URLs. Here you see HTML URL counts. Compare the previous file to the current one. Check the change rates. You can see all the outputs here. I won't go into detail now. I'll show you later. These are the settings for all our rules. Let me change the mode. Let's go into spider mode. You might not need many updates for a standard crawl. But say you have a test site. It's blocked by a robots.txt disallow command. You still want to crawl it.

Just go to the settings menu. Ignore the robots.txt file. Now you're ready to crawl. You can also set specific limits here if needed. Let me see what else to cover.

— Maybe I should mention robots.txt. There used to be a testing tool for it. I use this section in Scrum FROG often. You can add your robots.txt file. Then you can test any URL against it. Verify exactly which line blocks your URL. One wrong character can cost you many URLs. Scrum FROG has a built-in robots.txt testing tool. It works just like the classic one. If you have complex URL structures, test them here first. It’s the best way to move forward.

— Exactly. Great point. Thanks for jumping in. We talked about disavow tags. But sometimes pages have noindex or nofollow. If you need to crawl and analyze those, just check the nofollow box here. You can crawl them easily. What else? You can decide whether to crawl canonical tags, pagination, or hreflang tags. We discussed hreflang on the last podcast. You can manage all those settings here. If you want to include your sitemap in the crawl. Just add the sitemap URL here. This ensures all URLs in your sitemap get crawled.

— Sometimes sitemaps aren't in robots.txt. If you know your sitemaps, you can add them manually. Just use the section you clicked earlier. This forces the crawl to include those sitemaps.

— Aside from that, here.

— Exactly. You can configure JavaScript rendering right here. Enable it if your site relies on JavaScript. Don't worry if it seems slow. It might crawl at 0.25 URLs per second. It crawls slowly by design. That’s why JavaScript rendering packages in tools like Oncrawl or Lumar cost more. It creates extra overhead. Expect to wait a bit when using Scrum FROG locally. It will crawl eventually. Just know it takes longer. It has to render the JavaScript first.

— It’s not on by default. But Scrum FROG often prompts you to enable it. Don’t worry about it. Scrum FROG has excellent documentation. It shows you exactly what to do step-by-step. If you see a warning, just enable it in this tab. I’ll talk about the new AI features shortly. Those also require JavaScript rendering. You’ll need to keep this enabled. There are other advanced settings too. For example, the timeout is set to 20 by default. You can change that if needed. We can adjust these settings. Sometimes site response times vary. We might need to increase the timeout. We mentioned this earlier. In SERP mode, we can see title tags. It warns us if pixel counts exceed the limits.

You can change the default Scrum FROG settings for these pixels. For example, it flags anything under 30 characters as low. It also flags anything over 60 characters as an error. You can bump that to 65 if you want. You usually don't need to. But it's good to know where to update it. Anchor texts are also huge for SEO. It’s in English here. When we crawl, we can see overused anchor texts. Things like "click here," "go," or "here." You can update this list to filter the output you need.

For example, having too many links with "click here" isn't ideal. We don't want that. We want links using our actual target keywords. For instance, we want links for "red high heels" to use that exact phrase. That’s better for SEO. This section helps you track and analyze those anchor texts. Let's talk soft 404s. We'll cover this briefly now. We might touch on it again later. For those who don't know, here's the summary. A soft 404 is when a page returns a 200 status code. Even though the page is actually empty. It's basically a 404 disguised as a 200.

The server acts like the page loaded fine. But it's actually a "not found" error. Just like with anchor text. We can update or add soft 404 keywords for those pages. There's also a content section. Scrum FROG brings many features to this content section. It updates constantly. If you have duplicate pages, you can set the similarity percentage here. You can customize how it matches them. You can also check for grammar or spelling errors. I don't think it supports Turkish yet. No, it doesn't. Exactly. It's not there. It hasn't arrived for Turkish yet. But we've analyzed sites in other languages.

— It's been missing for years. I hope it gets added. There was a workaround, but it's not really worth the effort. You have to add something specific.

— It's a very serious process. Exactly.

— It should be default. There are other languages like Tamil. It would be great to have Turkish too. Maybe in the coming years.

— Right. There are many English versions. Why not Turkish?

— Exactly. Turkish would be awesome.

— True. But you can use it for foreign language sites.

— If you can't do it, just use AI. AI can handle it for you.

— Exactly. I wanted to dive into AI when I saw the embedding part. With AI updates, we use embeddings more. They understand the page structure and content. This helps us run better analyses. Enable this to unlock many AI features. Let me show you. Go to API access. I'll skip the basics here. You can connect Analytics and Search Console directly. Log in to your Google account. Select the account you need. You might have multiple properties there. It will show you a screen to pick one. Select your property to link everything.

Connect Analytics or Search Console. You also see third-party tools like Majestic, Ahrefs, and Moz. For OpenAI, it’s not just OpenAI. You can use Gemini and even Claude now. I’ll focus on OpenAI. The workflow is basically the same for all of them.

— You use it for a lot of tasks. I’ve seen your Basecamp messages. You use Scrum FROG to optimize content. It really makes your life easier.

— Absolutely. I love this part. We used to do this with Google Sheet. We used Apps Script for those tasks. For example, updating a meta description tag. Just paste your API key right here. Connect your account. Check out the existing library. There are plenty of prompts to explore. You can use them as is. Or tweak them to fit your needs. Want a custom prompt? Use AI to write one from scratch. Update titles and descriptions easily. Or build workflows to spot duplicate pages.

Tailor everything to your specific needs. There is also a feature here. Let me show you. You can generate images based on page content. They keep adding new features. Image generation wasn't there before. It’s worth keeping an eye on this. Test it out based on your workflow.

— Exactly. You can basically build your own mini-tool here. Just customize it to your needs.

— Totally. It works for any site. E-commerce or otherwise. You can generate alt text tags super fast. We used to do this with Google Sheet and APIs. Now you can do it much faster. We also talked about segmentation. You can run these tasks on specific segments. If I only want to update descriptions for the blog. I can just target that segment. Or if I want to create content for product categories. I can just select the category path segment. Then I can generate content specifically for those pages.

— I mentioned images earlier. Sorry to interrupt, but take alt text generation. Say you have a news page. The headline image is a Turkish soldier. That’s the current alt text. But Google has always said in SEO basics to describe the image. Something like "Turkish soldier holding a rifle on border duty" is better. Pinpointing the image content like that is super useful. Using these insights with your segmented plan can boost your image traffic. You can get bonus traffic from other channels. Using this feature actively is a big deal.

Great point.

— Exactly. Once you start, the rest just flows. You get hooked and want to do more. There are advanced settings here too. I won't dive too deep. You can reset or change your models as needed. There are other AI products too. You can use Claude. It’s not just for ChatGPT. You can use the free Gemini API. It’s quite limited, though. Free versions have their caps. Test it out first. Then use your own paid API key later. I skipped a few things in the AI section. Let me head back there. We covered robots.txt. Remember, in Spider mode, you don't have to crawl every single URL.

Do you want specific product paths? Maybe you want to include or exclude certain parameters. You can filter those paths using the include or exclude settings. They even provide examples. You might need different rules. For instance, use a slash followed by a dot and star. This crawls only URLs in that specific folder. Scrum FROG'un documentation covers these rules in detail.

— There’s a testing area once you add them. Just use the test URL feature. See if it’s actually excluded or included. It’s hard to make a mistake here. You can already see if it's correct.

— Manual testing is pretty much dead. We can use AI for everything now. It’s a huge help. I use exclude and include a lot. Especially for sites with many subfolders or subdomains. I actually use exclude more than include.

— Say we only want to crawl PDPs. If they have a pattern like "urun" in the URL, we just target those. We can build a strategy around that.

— Exactly.

— We crawled them and saved time. It took maybe half an hour. We delivered better work faster. It really frees up our schedule.

— Exactly. You can adjust the crawl speed here. I rarely change it. Sometimes you need to slow it down.

— Right. If you lack IP access or have user agent issues, clients set specific ones. Or they whitelist our IP. We can crawl using the company IP. If we aren't in a rush and the server is safe. I bump up the speed. If the server won't ban me, I'm good. If I have issues with a brand, I leave it alone. Or I lower the speed. I'd rather wait an extra half hour to get the job done.

— I just stop the crawl and handle other tasks.

— Exactly. Let it run in the background.

— Here is the user agent section Samet mentioned. It defaults to the Scrum FROG agent. You can select Googlebot desktop, smartphone, or even ChatGPT bot. This lets you crawl pages exactly like Googlebot would. You can also check Bing or DuckDuckGo. There are tons of details here. Let's talk about custom search and custom extraction. If you need to scan specific words in a site's HTML, use custom search. For example, if "out of stock" matters, you can find every page containing that phrase. You can also run specific scans using custom extraction.

Maybe you want to find and replace text under images. Just type your XPath right here. You can select either HTML or text formats. You can run multiple things at once. Just keep adding them as Extractor 1, 2, and so on. Once finished, check the custom extraction or custom search tabs to see your content. You'll find exactly what you're looking for there. Samet, maybe you can find a better example. I like that one.

— Exactly. I'll show it on screen. Let's talk about custom search. Say we search for "SEO" on our site. But "SEO" is in every title. Using custom search there isn't really an actionable strategy. What do we need here? We need to grab SEO terms in the article. Especially for internal linking. We can use custom extraction for this. I'll show you how to use it quickly. Let me show you. Here is the custom extraction part. You already showed this. I'll explain how to use it better. Let's promote my own blog post. I have a post about EAT. I'll select something from it.

Wait, I missed it. I'll select my name. I clicked it. The XPath appeared. It went to the team section. Sorry. Let me reopen it. You can grab the profile name. Or the author name. You can pull the breadcrumb. You can do anything here. Just hover over what you want. H1 is already there. I used the author profile as an example. Let me scroll down a bit. What else can I grab? I can grab anything. Let's stick to the author or date. I just want the dates. See, I selected it. It grabbed the class. This varies by site, by the way.

This is just an example from Zeo's site. If I add this as an extractor to Scrum FROG. It will give me this output after the crawl. It will show June 23, 2020. It will give me the date. I'll say okay. Here is my current strategy. I want to pull all publication dates from the Zeo blog. I can take action for this. I clicked okay. It output as HTML. I want it as text instead. I want it to show as June 23, 2020. Set it to text and click okay. You can even name it "Date." Aybüke mentioned this earlier. Adding too many extracts can clutter your crawl data.

It helps to label them for your spreadsheets or Excel exports. You need to know what is what. Maybe you just want the count or the values. You can do all of that. Finding the right extract used to be tough. It was hard for beginners. Now in Scrum FROG, you just click the element. It shows you the exact extract. I’d define it as a tool with tons of functional features.

— That was great. Really solid examples. It reminded me of something we use often. We frequently need to extract the category tree from headings. We use custom extraction a lot. Especially for stripping footer links. It’s a super active feature for us.

— Oh, absolutely. You can do literally anything with it. It’s endless. Let’s move on, Aybüke. Let’s talk response codes before we wrap up. There’s so much detail there. Like you just said. I’m deep in it too. It never ends. From bulk exports to AI with tasks. I want to cover response codes. They show the most common errors. Let me share my screen. It’ll be more helpful that way. Back in Scrum FROG. I crawled the Zeo site. I’m in response codes. What are they? They show if a page loads, redirects, or errors.

I’m filtering for redirects. I want to optimize crawl paths on Zeo. I need to update 301 links to their final URL. This saves time for Google bots and other crawlers. See all these URLs? If I send this to the team, there are 40 here. This is just an example. Filter for HTML pages too. Otherwise, you get PNGs and junk. Here is my first step. Start by analyzing response codes. I sort them by inlink count. Since our old team wrote blog posts, we see tons of links here.

Blog posts link to profiles. Prioritize these based on your site's needs. If a category has many links, start there. Optimizing one URL isn't the same as fixing thousands. Prioritization is key. My advice? Don't just dump 2,000 URLs on the dev team. Tell them you prioritized the top 300. It makes a huge difference. It’s easier for them to manage. Sort by inlinks and send it over. Take our HTTP status code guide. It has 4 links from a 301. Exporting the raw data is messy. We need the final URL to fix the source.

You get the redirect URL and the final URL. But where is the source? Click the URL. Go to the inlinks tab to see the source page. What’s the action plan? Find the source pages for these redirects. Swap the URL with the final URL. It’s just a simple lookup. Show the anchor text too. It helps clients pinpoint the exact issues. Always prioritize these details for redirects and response codes. Filter out external sources to make reports cleaner. Sometimes external links redirect or throw errors too.

Start with internal links. Then move to external ones. This is a common issue I see. I wanted to highlight it for you.

— That was great, Samet. You summarized the essentials perfectly. There is so much depth here. You can build a crawl for any need. They keep adding so many features. You can even calculate your digital footprint. It’s really unique.

— It’s packed with features. Use it right to build solid SEO strategies. It’s a great value for the price. Kudos to the developers. Check it out if you have international sites. You can check everything. We have tons of ideas right now. We could film hours of content on this. It is such a comprehensive tool.

— Exactly. You can check your schemas. Are you getting errors? Which pages have breadcrumb schemas? Which ones are missing them? You can spot all of that.

— Connect the Search Console API. We were discussing this with the team. When did Googlebot last visit a URL? You can check that. If it was ages ago, maybe your internal linking or click depth is off. You can run an analysis on that. It is vital for important pages. Say you have a thousand URLs. These are your top-selling pages. Take those thousand URLs as a sample. Or maybe 500. Run them in list mode in Scrum FROG. Connect the API. Get your click metrics. See why they aren't indexed. I wrote a post on crawled but not indexed pages recently.

Check that out too. Quality issues might have dropped your pages. You might see clicks for that URL. Maybe it had a thousand clicks. Then it dropped due to quality or other reasons. These are valuable insights for brands. You need to deep dive into Scrum FROG. Spend some time there.

— Exactly. You can do manual tasks with one click. If Google picks a different canonical, you'll see it. Just connect it to Search Console. You can check everything in bulk.

— Yeah, exactly.

— It’s worth exploring.

— Totally. Thanks for sharing, Aybüke. That’s all I had. Anything else you want to add?

— No, I think this was great. There are endless variations to discuss. This could go on forever. I think we’re good. Thanks to you too. It was fun.

— Exactly. I think this was a great video. Thanks to all our Zeo followers. Don't forget to follow us on social media. Take care. Goodbye.

— Goodbye.

This transcript was generated using AI speech recognition and verified against subtitle style rules.

Yiğit Konur

Content author

Yiğit Konur

both

Contact us

How we handle your data

Open as a full page