← Back to Feed
retoor
retoor · Level 55078
devlog

Sharding is not just for databases - shard your tests

I always thought sharding was a database thing.

You know the trick: sqlite3 has shit concurrency, but give each customer its own sqlite database and you suddenly scale better than the big boys. Domain sharding, technical sharding, whatever you want. You get the point.

But today AI taught me something new: you can shard TESTS too. Never seen this before and it is so fucking smart.

Problem: concurrent testing falls apart when tests touch data. You get flakes, locks, random failures, especially once you want to actually run the suite later in CI.

Solution I just designed: two shards.

  • Shard 1: tests that contain data. Run sequential. Safe, boring, correct.
  • Shard 2: everything else. Run extremely async. Fuck it, 48 playwright browsers at once. I have 64GB anyway.

Implementation is stupid simple: just add a decorator to your tests. That is all.

@shard(async=True) or @shard(async=False)

You can do whatever you want with it. Tag by domain, by speed, by flakiness, by browser need. The runner just splits and goes.

This project has a thousand tests so it actually matters. DevPlace has 4000 tests with no sharding yet, so I filed a ticket for that too.

What a time to be alive. Copy paste brain dump to Devii, ticket gets created, knowledge gets shared, tests go brrr.

Learn something new every day.

Comments

No comments yet. Start the discussion.