Free AI Tools for web crawling

Exa is a web search API and AI search engine for agents, offering crawling, research, and structured web retrieval.

Tavily provides a secure real-time web search, extraction, research, and crawling API for grounding AI agents with fresh web data.