Fetching Historic Payruns with rate limit

1

Hey there :slight_smile:

I’m an engineer from Penfold. We’re a pension provider and have a FreeAgent integration for our customers to link their FreeAgent account. We are looking to fetch filled pay runs over the span of 2 years or so. Unfortunately, due to the rate limits (as per the documentation) we’re not able to do this without hitting the limits and need to do this for a substantial amount of employers. We’re also having to hit these endpoints:

  • GET /api.freeagent.com/v2/payroll/{year}/{period}

  • GET /api.freeagent.com/v2/users/{user_id}

  • GET /api.freeagent.com/v2/payroll_profiles/{year}?user={user_url}

    Is there any way we could get this data more efficiently/reliably than having to call this endpoint repeatedly? If possible at all, we’d ideally appreciate if this data can be sent to us :folded_hands:

    I have already reached out to: support@freeagent.com but they pointed me here instead.

    Any help would be greatly appreciate it! Thank you!

2

Hi Emilios, Note sure of your exact workflow, but I assume you mean your getting the data, just not as fast as you or your end clients wish?

Per my understanding of the free agent documentation the rate limits are per user account (individual user) and that does not stop you from extracting the data, it controls how much data you can get out per minute/hour. Standard for cloud apps with shard tenancy and stop ddos attacks etc.

Try 1 or more of:

  1. batching your requests: GET say x records at a time (say x = 200 per Request)

  2. (more advanced): A distributed map cache (redis or valkey etc) for repetitive calls, ie store data there that does not change frequently to save freeagent repetitive calls to eliminate rate limit issues. EG: You poll free agent once or twice a day and store the data in a distrubuted map cache, you then call that map cache as many times as you want, ie up to 100,000s of times per second etc. Redis benchmark | Docs

  3. Workaround, per my above first comment, api limits are per api token, create a second api account to effectively double your rate limits,

Cheers

Cian

3

Hi Cian,

Thanks for you input :slight_smile:

Unfortunately, that’s not quite our issue. It’s more that we are trying to fetch historic data: up to 2 years ago. Due the rate limitations (which are normal/reasonable) it would take us quite a while.

Our goal is to ideally get some help from FreeAgent with them potentially providing this information for us hopefully.

Emilios