hi there! this should include everything you need to know about gloogle. if you have any questions, feel free to dm me on twitter, i'm @tiagozip_
i have my own little spider go around the web clicking every link it finds, collecting some info about it, checking if it's okay to scrape, and add it to my db.
then, sqlite adds it to its full text search index, which is then used for the actual searching. i might try to use embeddings in the future, but for now, this is good enough.
the crawler is in maintenance mode right now. it's running at a significantly slower and calmer speed, mostly just updating existing records. this lets me focus on other stuff and not use as much resources.
yes! new pages are constantly being indexed every day. currently its being ran in a kinda stupid way, i have a cron job that runs the crawler for 1 hour ever day at around 3:06 am to not affect other resources.
glooglecrawler respects the X-Robots-Tag header if it includes noindex, nofollow or nosnippet. noarchive and noimageindex don't apply since glooglecrawler does not index images or generate page archives.
all glooglecrawler requests are also marked with an useragent similar to "Mozilla/5.0 (compatible; GloogleCrawler/... +https://...)". links with nofollow aren't followed too.
we might send some snippets to local and cloud llms for moderation, but they won't train on your content.
any form of nsfw or content illegal in portugal or the european union, hate speech, extremism, phishing and malware, and doxxing
everything else should be allowed but can be taken down for any reason. if something that isn't public domain is indexed but legally can't, please let me know
for the search ui, we're using bun, elysia and sqlite. i'm planning to try out semantic search soon maybe.
for the crawler, just bun and node-html-parser. that's it.
the crawler and ui isn't open-source. the initial 1M dataset is available for download on zenodo, but live newer updates aren't.
you're allowed to use gloogle however you like, as long as you don't spam, abuse, or hammer it with too many automated requests. if you're building something on top of it, follow reasonable limits.
you are fully allowed to reverse-engineer our api. we use cloudflare for ddos protection.