Skip to main content
data.metadata is an object on every successful scrape. A field is left out when the page does not have it, so check that it is there before you use it. A page with no meta tags still has sourceURL, url, statusCode, contentType and cacheState.
Send "allMetaTags": true for every other tag too: ogImage, favicon, the Open Graph and Dublin Core fields under plain names such as ogTitle and dcDate, and every meta tag under its own name, such as og:title or twitter:card. A tag that appears several times comes back as an array of strings. A YouTube link has no meta tags to read. Its title and description come from the video, and ogImage with allMetaTags. A page that answers 404 or 410 has that statusCode and an error, and costs nothing. Crawl returns the same object for each crawled page.