The User-agent line in a robots.txt file identifies which specific crawler a block of rules applies to, Googlebot, Bingbot, or a specific tool's crawler, since different bots can be given different permissions within the same file. User-agent: * is the wildcard that applies a rule set to every crawler that doesn't have its own specific block.
That per-bot targeting is useful for more than blanket allow/block decisions: a site might want to permit Googlebot broadly while blocking a specific SEO tool's crawler from hammering the server with requests, or apply different crawl rules to an image-specific bot versus the main web crawler. Each User-agent line starts a new block of directives that apply only to the bot named, until the next User-agent line starts a new block, so the order and grouping of a robots.txt file matters for which rules actually apply to which crawler. Getting this structure wrong, rules meant for one bot accidentally bleeding into the wildcard block, or vice versa, is a common source of unintended crawl blocks that are hard to spot without carefully reading the file's actual grouping.
