How to Scrape Mastodon with Go: Profiles, Posts, Hashtags Across Servers, and Trends (2026)
Scraping Mastodon is unusual in a good way: you don't need to parse HTML at all. Every Mastodon server exposes the same public REST API, and most read endpoints answer without a token.
The hard part isn't getting data. It's that there is no single "Mastodon". There are thousands of servers, each with its own partial view of the network, its own rate limits, and its own settings.
In this post I'll build a small Mastodon scraper in Go, standard library only, that:
- Looks up a profile on the account's home server
- Pages through a user's original posts for the last N days and computes engagement
- Collects a hashtag from several servers and de-duplicates the results
- Reads trending hashtags per server
- Writes everything to CSV
Everything below was run on October 26, 2026, and the output is real.
Before you start: read the server's rules and terms, keep request rates low, and respect people's choices. Mastodon accounts can set noindex (opt out of search engines) and indexable flags; the code below skips noindex accounts. Don't build profiles of individual people, and don't republish their posts out of context.
How the Mastodon API looks from a scraper's point of view
All the endpoints in this post are plain GET requests that return JSON:
- Profile by handle -
/api/v1/accounts/lookup?acct=user- No Auth? - A user's posts -
/api/v1/accounts/:id/statuses- No - Hashtag timeline -
/api/v1/timelines/tag/:tag- Usually no - Trending tags -
/api/v1/trends/tags- No - Public/federated timeline -
/api/v1/timelines/public- Often yes - Full-text post search -
/api/v2/search?type=statuses- Yes
The last two rows are the first surprise. On mastodon.social, the public timeline answers HTTP 422 with {"error":"This method requires an authenticated user"}, and status search returns an empty statuses array when you're logged out. Server admins can turn these on or off, so check them on each server you target.
Pagination isn't in the JSON body. It's in the Link response header, like GitHub's API:
Link: <https://mastodon.social/api/v1/accounts/1/statuses?max_id=115...>; rel="next", <...>; rel="prev"
Rate limits are in headers too. Unauthenticated requests on mastodon.social got x-ratelimit-limit: 300 per five-minute window, with x-ratelimit-remaining and an x-ratelimit-reset timestamp.
Step 1: A polite HTTP helper
mkdir mastoscraper && cd mastoscraper
go mod init mastoscraper
The helper sends a real User-Agent (put your own contact URL in it), retries on network errors and 5xx, sleeps until the reset time on a 429, and slows down on its own when fewer than 20 requests are left in the window. It returns the response headers, because we need Link for pagination.
package main
import (
"encoding/csv"
"encoding/json"
"errors"
"flag"
"fmt"
"html"
"net/http"
"net/url"
"os"
"regexp"
"sort"
"strconv"
"strings"
"time"
)
const userAgent = "mastoscraper/0.1 (+https://example.com/contact)"
var client = & http . Client { Timeout : 20 * time . Second }
// getJSON fetches a URL, decodes JSON into out, and returns the response
// headers so callers can read Link (pagination) and X-RateLimit-*.
func getJSON ( u string , out any ) ( http . Header , error ) {
for attempt := 0 ; attempt < 4 ; attempt ++ {
req , _ := http . NewRequest ( "GET" , u , nil )
req . Header . Set ( "User-Agent" , userAgent )
req . Header . Set ( "Accept" , "application/json" )
res , err := client . Do ( req )
if err != nil {
time . Sleep ( time . Duration ( attempt + 1 ) * 2 * time . Second )
continue }
if res . StatusCode == 429 {
wait := untilReset ( res . Header )
fmt . Fprintf ( os . Stderr , "429 from %s, sleeping %s \n " , res . Request . URL . Host , wait )
res . Body . Close ()
time . Sleep ( wait )
continue }
if res . StatusCode >= 500 {
res . Body . Close ()
time . Sleep ( time . Duration ( attempt + 1 ) * 3 * time . Second )
continue }
if res . StatusCode != 200 {
res . Body . Close ()
return res . Header , fmt . Errorf ( "GET %s: HTTP %d" , u , res . StatusCode )
}
err = json . NewDecoder ( res . Body ) . Decode ( out )
res . Body . Close ()
// Be polite even when we have budget left.
if rem , _ := strconv . Atoi ( res . Header . Get ( "X-RateLimit-Remaining" )); rem > 0 && rem < 20 {
time . Sleep ( untilReset ( res . Header ))
}
return res . Header , err }
return nil , errors . New ( "gave up after retries: " + u )
}
func untilReset ( h http . Header ) time . Duration {
if t , err := time . Parse ( time . RFC3339Nano , h . Get ( "X-RateLimit-Reset" )); err == nil {
if d := time . Until ( t ); d > 0 {
return d + time . Second }
}
return 30 * time . Second }
// nextURL pulls the rel="next" URL out of a Link header like
// <https://host/api/v1/...?max_id=123>; rel="next", <...>; rel="prev"
var linkNext = regexp . MustCompile ( `<([^>]+)>;\s*rel="next"` )
func nextURL ( h http . Header ) string {
m := linkNext . FindStringSubmatch ( h . Get ( "Link" ))
if m == nil {
return "" }
return m [ 1 ] }
Step 2: Types and small helpers
Mastodon returns post bodies as HTML (<p>, links, mention spans), so plainText strips tags and unescapes entities. splitHandle accepts @user@server, user@server, or a profile URL.
type Account struct {
ID string `json:"id"`
Acct string `json:"acct"`
DisplayName string `json:"display_name"`
URL string `json:"url"`
FollowersCount int `json:"followers_count"`
FollowingCount int `json:"following_count"`
StatusesCount int `json:"statuses_count"`
LastStatusAt string `json:"last_status_at"`
Bot bool `json:"bot"`
Indexable * bool `json:"indexable"`
Noindex * bool `json:"noindex"`
}
type Status struct {
ID string `json:"id"`
URI string `json:"uri"`
URL string `json:"url"`
CreatedAt time . Time `json:"created_at"`
Content string `json:"content"`
Language string `json:"language"`
RepliesCount int `json:"replies_count"`
ReblogsCount int `json:"reblogs_count"`
FavouritesCount int `json:"favourites_count"`
Account Account `json:"account"`
Reblog * Status `json:"reblog"`
Tags [] struct {
Name string `json:"name"`
} `json:"tags"`
}
var tagRe = regexp . MustCompile ( `<[^>]+>` )
func plainText ( h string ) string {
h = strings . ReplaceAll ( h , "</p><p>" , " \n\n " )
h = strings . ReplaceAll ( h , "<br />" , " \n " )
return strings . TrimSpace ( html . UnescapeString ( tagRe . ReplaceAllString ( h , "" )))
}
// splitHandle turns "@user@server", "user@server" or a profile URL into (user, server).
func splitHandle ( handle string ) ( string , string , error ) {
if strings . HasPrefix ( handle , "https://" ) {
u , err := url . Parse ( handle )
if err != nil {
return "" , "" , err }
return strings . TrimPrefix ( strings . Trim ( u . Path , "/" ), "@" ), u . Host , nil }
parts := strings . Split ( strings . TrimPrefix ( handle , "@" ), "@" )
if len ( parts ) != 2 {
return "" , "" , fmt . Errorf ( "want user@server, got %q" , handle )
}
return parts [ 0 ], parts [ 1 ], nil }
Step 3: Profiles and posts, from the home server
Any server can look up any account it knows about. mastodon.social will happily answer lookup?acct=g*****@hachyderm.io. But that answer is mastodon.social's cached copy, and it's only as fresh as the last time mastodon.social refreshed it.
In my checks today the numbers matched, but nothing guarantees that, and copies of posts on other servers definitely lag behind on favourites and boosts. So the code asks the account's own server.
exclude_replies=true&exclude_reblogs=true keeps only original posts, limit=40 is the maximum page size, and the loop stops when posts get older than the cutoff. Posts come back newest first, so it can stop early instead of paging through years of history.
// lookup asks the account's HOME server, not mastodon.social, so the counts are authoritative.
func lookup ( handle string ) ( Account , string , error ) {
user , server , err := splitHandle ( handle )
if err != nil {
return Account {}, "" , err }
var a Account _ , err = getJSON ( fmt . Sprintf ( "https://%s/api/v1/accounts/lookup?acct=%s" , server , url . QueryEscape ( user )), & a )
return a , server , err }
func userPosts ( handle string , maxPages int , since time . Time ) ([] Status , error ) {
a , server , err := lookup ( handle )
if err != nil {
return nil , err }
if a . Noindex != nil && * a . Noindex {
return nil , fmt . Errorf ( "%s has opted out of search indexing (noindex); skipping" , handle )
}
next := fmt . Sprintf ( "https://%s/api/v1/accounts/%s/statuses?limit=40&exclude_replies=true&exclude_reblogs=true" , server , a . ID )
var all [] Status
for page := 0 ; next != "" && page < maxPages ; page ++ {
var batch [] Status
h , err := getJSON ( next , & batch )
if err != nil {
return all , err }
stop := false
for _ , s := range batch {
if s . CreatedAt . Before ( since ) {
stop = true
break }
all = append ( all , s )
}
if stop || len ( batch ) == 0 {
break }
next = nextURL ( h )
}
return all , nil }
$ ./mastoscraper profile @G******@mastodon.social
Eugen Rochko (Gargron) on mastodon.social
followers 383376 · following 764 · posts 82664 · last post 2026-10-26 · bot=false
$ ./mastoscraper posts -days 30 -pages 3 @G******@mastodon.social
34 original posts in the last 30 days
avg favourites 184.7 · avg boosts 61.3
1012 ★ 451 ⟳Heh. I misremembered the date--thought it was today, but loo…
961 ★ 450 ⟳"AI" is a failed technology, in the sense that it's neither …
468 ★95 ⟳#SilentSunday
Note that last_status_at is a date, not a timestamp. Mastodon rounds it on purpose.
Step 4: One hashtag, several servers, no duplicates
This is where federation matters. A hashtag timeline on a server only contains posts that server has seen: posts by its own users, plus posts from accounts someone on that server follows. #golang on mastodon.social, fosstodon.org, and hachyderm.io are three different, overlapping lists.
So the code reads the tag from each server and de-duplicates by uri. The uri is the post's canonical ActivityPub id, and it's the same on every server. The id field is not : each server assigns its own local id to its copy of a post. When the same post shows up more than once, it keeps the copy with the highest counts, since remote copies are the ones that lag.
// tagAcrossServers reads the same hashtag from several servers and de-duplicates by URI,
// because each server only sees the posts that federated to it.
func tagAcrossServers ( tag string , servers [] string , maxPages int ) ( map [ string ] Status , map [ string ] int ) {
seen := map [ string ] Status {}
perServer := map [ string ] int {}
for _ , srv := range servers {
next := fmt . Sprintf ( "https://%s/api/v1/timelines/tag/%s?limit=40" , srv , url . PathEscape ( tag ))
for page := 0 ; next != "" && page < maxPages ; page ++ {
var batch [] Status
h , err := getJSON ( next , & batch )
if err != nil {
fmt . Fprintf ( os . Stderr , "%s: %v \n " , srv , err )
break }
for _ , s := range batch {
perServer [ srv ] ++
prev , ok := seen [ s . URI ]
// Keep the copy with the highest counts; remote copies lag behind.
if ! ok || s . FavouritesCount + s . ReblogsCount > prev . FavouritesCount + prev . ReblogsCount {
seen [ s . URI ] = s }
}
if len ( batch ) == 0 {
break }
next = nextURL ( h )
}
}
return seen , perServer }
The acct field has its own trap. It's relative to the server you asked: an account local to hachyderm.io shows up as plain golang when you ask hachyderm.io, and as g*****@hachyderm.io everywhere else. If you mix servers, normalize it:
// fullAcct makes "user" (local to whichever server we asked) into "user@server".
func fullAcct ( a Account ) string {
if strings . Contains ( a . Acct , "@" ) {
return a . Acct }
if u , err := url . Parse ( a . URL ); err == nil {
return a . Acct + "@" + u . Host }
return a . Acct }
func short ( s string , n int ) string {
s = strings . ReplaceAll ( s , " \n " , " " )
if r := [] rune ( s ); len ( r ) > n {
return string ( r [ : n ]) + "…" }
return s }
$ ./mastoscraper tag -pages 2 -out golang.csv golang mastodon.social80 posts fosstodon.org80 posts hachyderm.io 80 posts
240 posts fetched, 106 unique after de-duplicating by URI
34 ★ re*********@masto.astropirad… Bueno, mastodones. Hoy por sorpresa y sin dar ni l…
27 ★ g*****@hachyderm.io🥳 Go 1.27.3 and 1.26.10 are released!🔏 Security:…
7 ★ om********@mastodon.social Dear #fediverse,Introducing gracias, a #noai cli…
6 ★ s******@hachyderm.io Lock down your website to a dedicated few browsers…
5 ★ ha********@oldbytes.spaceThe BathroomOriginally by Matthew Smith.#zenpl…
240 posts fetched, 106 unique. More than half of what you download from a second and third server is stuff you already have. That's normal, and it's why de-duplicating by uri isn't optional.
Also notice that the top post is in Spanish and not really about Go: hashtags are free text, so filter by language and your own keywords if you need precision.
Step 5: Trending hashtags per server
/api/v1/trends/tags returns each tag with a history array of daily uses and accounts. The first trap: history[0] is today, which is still being counted. My first run printed #tatort (0) and #FineArtAPlaceOrDestination (218) from today's partial numbers; switching to history[1] (yesterday, a complete day) gives comparable numbers. A tag can also trend today and still show 0 for yesterday, as you'll see below.
type TrendTag struct {
Name string `json:"name"`
History [] struct {
Day string `json:"day"`
Uses string `json:"uses"`
Accounts string `json:"accounts"`
} `json:"history"`
}
$ ./mastoscraper trends mastodon.social#SilentSunday (884), #audi (189), #OppositesAMovie (713), #FineArtAPlaceOrDestination (0), #tatort (72) fosstodon.org#SilentSunday (696), #Monsterdon (25), #Serbia (179), #CrispSandwichDay (51), #Tatort (31) hachyderm.io #silentsunday (717), #OppositesAMovie (641), #audi (113), #FineArtAPlaceOrDestination (0), #my9albums (45)
Trends are per server and moderated: admins review tags before they can trend. So "trending on mastodon.social" isn't "trending on Mastodon", and case varies (#SilentSunday vs #silentsunday). Lowercase tag names before you compare servers.
Comments
No comments yet. Start the discussion.