Shipped Something New 👀

Okkayyy so shipped a new component in Agentgenesis after a looooong time 😭✨

This time it’s the TweetScraper Tool — a handy little tool you can use to scrape tweets from accounts and also extract text from specific tweets. 🐦📤

Internally, I’ve used some magical Apify actors for this. And if you’re wondering...

What Even Is Apify?

So Apify is basically a marketplace of services (called actors) where people upload scrapers and automation tools, and you can use them — some are paid, some are freemium.

Now after some digging, I figured out two 🔥 actors that:

can extract tweets from user accounts,
extract text from individual tweets,
and... they’ve been working consistently for a long time, super stable,
cheap af — you can probably do most of your scraping with just the free $5 monthly credits you get.

Isn’t This Just a Wrapper?

Yeah, technically this is a wrapper around those Apify actors... BUT here's why it’s more than that:

You don’t need to spend hours figuring out which actor to use.
You don’t need to learn how to plug in and integrate with Apify.
You don’t even need to worry about stability or pricing.

All that headache is gone. Wrapped up nicely inside this TweetScraperTool class. It’s expandable, time-saving, and makes your life easier. So yeah — a wrapper, but one that actually slaps. 💥

How To Use 🛠️

Read the docs once on how to set it up:
👉 Installation Docs

Scraping Single Tweet Text

You can scrape a tweet using the tweet ID:

const singleTweetData = await tweetScraper.scrapeTweetText({
  tweetId: "1944299394032882148",
});

Or with the tweet URL:

const singleTweetData = await tweetScraper.scrapeTweetText({
  tweetUrl: "https://x.com/kirat_tw/status/1943192256497762309",
});

Both options work fine — whether you provide the ID or full URL. 🙌

Scraping Multiple Latest Tweets from an Account

If you wanna scrape recent tweets from a user:

const latestTweets = await tweetScraper.scrapeTweets("https://x.com/mannupaaji", 5);

Parameters:

accountUrl — URL of the Twitter account
count — (optional) how many tweets to fetch (defaults to 10)
sinceDate — (optional) start date to fetch tweets from (defaults to 2024-03-05)

Wanna try the sinceDate param in action? Let's say you want to scrape 5 tweets from 2 days ago, you can do:

const twoDaysAgo = new Date();
twoDaysAgo.setDate(twoDaysAgo.getDate() - 2);

const tweets = await tweetScraper.scrapeTweets("https://x.com/mannupaaji", 5, twoDaysAgo);

This will start fetching tweets posted from 2 days earlier — super handy when you’re working with fresh timelines or filtering recent buzz. 🔍

So That’s It 😎

Copy-paste, experiment, extend the code... go build something cool.
And hey —

Tag me on Twitter 🐤
Drop a ⭐ on GitHub 🚀

Happy Tweet Scraping!