API Result Caching

Hai there,

The guys who run the Freeagent twitter account told me to drop on down
here and ask about caching.

I’m working on a system that uses the Freeagent API will be doing a
fair few API calls.

My initial thought was to use memcache but because of the size of the
data I don’t think that’d be appropriate.
My second thought was to save the results as xml files and set an
expiry time within the code for each type of call, and obviously when
the data was updated via my system.

Just wanted to know if you guys have any better thoughts on it, like
methods you’ve used.

Regards
Ollie

The original plan was for the system to be a personal one, but many
people have shown interest and I’ve contacted the respective parties
about the partnership agreements that they have allowing users to sign
up for both apps through my own.

It could potentially be 100’s and 1000’s of users, with all of the
data from them obviously accumulating over time.

I have both options, MySQL and MongoDB options available.On Oct 17, 2:36 pm, Nic Wise n...@fastchicken.co.nz wrote:

Ahaha do people actually use ‘webscale’ as a term? New one to me!

I guess some do, or the meem would never have been created.

I don’t feel overly comfortable writing masses of data to MySQL like
the option above, I really wish I could tell you why but I don’t know.
I imagine that at some point in time I had a reason but it’s just
become far too normal for me to not use it now.

How much data are you looking at?A few 100meg? or is this for 100’s or
1000’s of customers?

I’m just quite the fan of MongoDB. I think I’ll do some benchmarking
for the particular data in question and see what results it yields.

I’m a fan of nosql in general (RavenDB, in my case, being a .NET
person). Works really well in some cases. And this is very key-value
oriented (key: url. value: xml) so would be good for quick storage.

Guess it depends what you have available.

On Oct 17, 12:09 pm, Nic Wise n...@fastchicken.co.nz wrote:

hehe, it was more a pi**take at mongodb

http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale

(which, if setup and done correctly, IS “webscale”, tho anyone using
that word for real should have a baseball bat taken to their head)

On Mon, Oct 17, 2011 at 12:04, Olly o...@freeagent.com wrote:

On Monday, 17 October 2011 12:01:15 UTC+1, Nic Wise wrote:

Of course, MySql isn’t webscale…

Yeah, right.
http://highscalability.com/blog/2011/7/18/new-relic-architecture-coll

7 Billion new rows of data every day. Data collection handled by 9 sharded
MySQL servers

Olly


You received this message because you are subscribed to the Google Groups
“FreeAgent API” group.
To view this discussion on the web visit
https://groups.google.com/d/msg/freeagent_api/-/3X72ppV8dNIJ.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to
freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/freeagent_api?hl=en.


Nic Wise
t. +44 7788 592 806 | @fastchicken |http://www.linkedin.com/in/nicwise
b.http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop.http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go:http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding!http://goo.gl/Icp2


You received this message because you are subscribed to the Google Groups “FreeAgent API” group.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/freeagent_api?hl=en.


Nic Wise
t. +44 7788 592 806 | @fastchicken |http://www.linkedin.com/in/nicwise
b.http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop.http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go:http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding!http://goo.gl/Icp2

The project in question is a web app, written in PHP.

As it is I’m using MySQL, but if I were to cache the results in the
database, I’d look at moving to my other server to utilise MongoDB.

The idea of the app is to pull through functionality from FA and
seamlessly merge with that of another system, making life simpler.On Oct 17, 8:47 am, Nic Wise n...@fastchicken.co.nz wrote:

BTW, I would suggest that you (for example)

On start, load contacts and projects and tasks. Keep them in memory somewhere

Do a run. Don’t re-request C, P or T again, use the in-mem ones

After the run (if it’s short), dump them, and reload on the next one

That keeps it up to date, but without slamming the FA api service.
Should be quicker too.

But some more context around what you are going would help.

On Mon, Oct 17, 2011 at 01:12, Ollie m...@ollieread.com wrote:

Hai there,

The guys who run the Freeagent twitter account told me to drop on down
here and ask about caching.

I’m working on a system that uses the Freeagent API will be doing a
fair few API calls.

My initial thought was to use memcache but because of the size of the
data I don’t think that’d be appropriate.
My second thought was to save the results as xml files and set an
expiry time within the code for each type of call, and obviously when
the data was updated via my system.

Just wanted to know if you guys have any better thoughts on it, like
methods you’ve used.

Regards
Ollie


You received this message because you are subscribed to the Google Groups “FreeAgent API” group.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/freeagent_api?hl=en.


Nic Wise
t. +44 7788 592 806 | @fastchicken |http://www.linkedin.com/in/nicwise
b.http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop.http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go:http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding!http://goo.gl/Icp2

Ahaha do people actually use ‘webscale’ as a term? New one to me!

I guess some do, or the meem would never have been created.

I don’t feel overly comfortable writing masses of data to MySQL like
the option above, I really wish I could tell you why but I don’t know.
I imagine that at some point in time I had a reason but it’s just
become far too normal for me to not use it now.

How much data are you looking at?A few 100meg? or is this for 100’s or
1000’s of customers?

I’m just quite the fan of MongoDB. I think I’ll do some benchmarking
for the particular data in question and see what results it yields.

I’m a fan of nosql in general (RavenDB, in my case, being a .NET
person). Works really well in some cases. And this is very key-value
oriented (key: url. value: xml) so would be good for quick storage.

Guess it depends what you have available.> On Oct 17, 12:09 pm, Nic Wise n...@fastchicken.co.nz wrote:

hehe, it was more a pi**take at mongodb

http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale

(which, if setup and done correctly, IS “webscale”, tho anyone using
that word for real should have a baseball bat taken to their head)

On Mon, Oct 17, 2011 at 12:04, Olly o...@freeagent.com wrote:

On Monday, 17 October 2011 12:01:15 UTC+1, Nic Wise wrote:

Of course, MySql isn’t webscale…

Yeah, right.
http://highscalability.com/blog/2011/7/18/new-relic-architecture-coll

7 Billion new rows of data every day. Data collection handled by 9 sharded
MySQL servers

Olly


You received this message because you are subscribed to the Google Groups
“FreeAgent API” group.
To view this discussion on the web visit
https://groups.google.com/d/msg/freeagent_api/-/3X72ppV8dNIJ.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to
freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/freeagent_api?hl=en.


Nic Wise
t. +44 7788 592 806 | @fastchicken |http://www.linkedin.com/in/nicwise
b.http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop.http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go:http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding!http://goo.gl/Icp2


You received this message because you are subscribed to the Google Groups “FreeAgent API” group.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/freeagent_api?hl=en.

Nic Wise
t. +44 7788 592 806 | @fastchicken | http://www.linkedin.com/in/nicwise
b. http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop. http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.
http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go: http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding! http://goo.gl/Icp2

BTW, I would suggest that you (for example)

On start, load contacts and projects and tasks. Keep them in memory somewhere

Do a run. Don’t re-request C, P or T again, use the in-mem ones

After the run (if it’s short), dump them, and reload on the next one

That keeps it up to date, but without slamming the FA api service.
Should be quicker too.

But some more context around what you are going would help.On Mon, Oct 17, 2011 at 01:12, Ollie me@ollieread.com wrote:

Hai there,

The guys who run the Freeagent twitter account told me to drop on down
here and ask about caching.

I’m working on a system that uses the Freeagent API will be doing a
fair few API calls.

My initial thought was to use memcache but because of the size of the
data I don’t think that’d be appropriate.
My second thought was to save the results as xml files and set an
expiry time within the code for each type of call, and obviously when
the data was updated via my system.

Just wanted to know if you guys have any better thoughts on it, like
methods you’ve used.

Regards
Ollie


You received this message because you are subscribed to the Google Groups “FreeAgent API” group.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/freeagent_api?hl=en.

Nic Wise
t. +44 7788 592 806 | @fastchicken | http://www.linkedin.com/in/nicwise
b. http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop. http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.
http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go: http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding! http://goo.gl/Icp2

How about storing the results in a db table (wither that be as raw
results, or as a json array, depends on what you’re pulling back) ,
then set an expiry date so that once the date has passed, recall the
API and update the table.On Oct 17, 1:12 am, Ollie m...@ollieread.com wrote:

Hai there,

The guys who run the Freeagent twitter account told me to drop on down
here and ask about caching.

I’m working on a system that uses the Freeagent API will be doing a
fair few API calls.

My initial thought was to use memcache but because of the size of the
data I don’t think that’d be appropriate.
My second thought was to save the results as xml files and set an
expiry time within the code for each type of call, and obviously when
the data was updated via my system.

Just wanted to know if you guys have any better thoughts on it, like
methods you’ve used.

Regards
Ollie

Ahaha do people actually use ‘webscale’ as a term? New one to me!

I don’t feel overly comfortable writing masses of data to MySQL like
the option above, I really wish I could tell you why but I don’t know.
I imagine that at some point in time I had a reason but it’s just
become far too normal for me to not use it now.

I’m just quite the fan of MongoDB. I think I’ll do some benchmarking
for the particular data in question and see what results it yields.On Oct 17, 12:09 pm, Nic Wise n...@fastchicken.co.nz wrote:

hehe, it was more a pi**take at mongodb

http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale

(which, if setup and done correctly, IS “webscale”, tho anyone using
that word for real should have a baseball bat taken to their head)

On Mon, Oct 17, 2011 at 12:04, Olly o...@freeagent.com wrote:

On Monday, 17 October 2011 12:01:15 UTC+1, Nic Wise wrote:

Of course, MySql isn’t webscale…

Yeah, right.
http://highscalability.com/blog/2011/7/18/new-relic-architecture-coll

7 Billion new rows of data every day. Data collection handled by 9 sharded
MySQL servers

Olly


You received this message because you are subscribed to the Google Groups
“FreeAgent API” group.
To view this discussion on the web visit
https://groups.google.com/d/msg/freeagent_api/-/3X72ppV8dNIJ.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to
freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/freeagent_api?hl=en.


Nic Wise
t. +44 7788 592 806 | @fastchicken |http://www.linkedin.com/in/nicwise
b.http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop.http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go:http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding!http://goo.gl/Icp2

The project in question is a web app, written in PHP.

As it is I’m using MySQL, but if I were to cache the results in the
database, I’d look at moving to my other server to utilise MongoDB.

Why? Just have a table with an id, a url and a text blob. And maybe an
expiry time :slight_smile: Easy.

Of course, MySql isn’t webscale…

The idea of the app is to pull through functionality from FA and
seamlessly merge with that of another system, making life simpler.

sounds good. So, it’s running in the background, on a schedule? Or
when someone does something on the webpage?

Just trying to get a handle on if you have to throw everything away
after each run (ie, stateless web model) or you can keep things in
memory between “runs” (stateful service / daemon model)

APC cache or memcached sound like the best options, IMO. You want
fresh data, but at the same time, you dont want to slam FA… sounds
like a good middle ground.

That said, my iphone app downloads a good chunk of someone’s FA
account each time it syncs, and thats not usually a problem… eg
once an invoice is closed, you can’t usually re-open them, so you
don’t need to look at changes for those.

BTW, I would suggest that you (for example)

On start, load contacts and projects and tasks. Keep them in memory somewhere

Do a run. Don’t re-request C, P or T again, use the in-mem ones

After the run (if it’s short), dump them, and reload on the next one

That keeps it up to date, but without slamming the FA api service.
Should be quicker too.

But some more context around what you are going would help.

Hai there,

The guys who run the Freeagent twitter account told me to drop on down
here and ask about caching.

I’m working on a system that uses the Freeagent API will be doing a
fair few API calls.

My initial thought was to use memcache but because of the size of the
data I don’t think that’d be appropriate.
My second thought was to save the results as xml files and set an
expiry time within the code for each type of call, and obviously when
the data was updated via my system.

Just wanted to know if you guys have any better thoughts on it, like
methods you’ve used.

Regards
Ollie


You received this message because you are subscribed to the Google Groups “FreeAgent API” group.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/freeagent_api?hl=en.


Nic Wise
t. +44 7788 592 806 | @fastchicken |http://www.linkedin.com/in/nicwise
b.http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop.http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go:http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding!http://goo.gl/Icp2


You received this message because you are subscribed to the Google Groups “FreeAgent API” group.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/freeagent_api?hl=en.

Nic Wise
t. +44 7788 592 806 | @fastchicken | http://www.linkedin.com/in/nicwise
b. http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop. http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.
http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go: http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding! http://goo.gl/Icp2

Of course, MySql isn’t webscale…

Yeah, right.

  • 7 Billion new rows of data every day. Data collection handled by 9
    sharded MySQL servers*

Olly

hehe, it was more a pi**take at mongodb

http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale

(which, if setup and done correctly, IS “webscale”, tho anyone using
that word for real should have a baseball bat taken to their head)On Mon, Oct 17, 2011 at 12:04, Olly olly@freeagent.com wrote:

On Monday, 17 October 2011 12:01:15 UTC+1, Nic Wise wrote:

Of course, MySql isn’t webscale…

Yeah, right.
http://highscalability.com/blog/2011/7/18/new-relic-architecture-collecting-20-billion-metrics-a-day.html

7 Billion new rows of data every day. Data collection handled by 9 sharded
MySQL servers

Olly


You received this message because you are subscribed to the Google Groups
“FreeAgent API” group.
To view this discussion on the web visit
https://groups.google.com/d/msg/freeagent_api/-/3X72ppV8dNIJ.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to
freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/freeagent_api?hl=en.

Nic Wise
t. +44 7788 592 806 | @fastchicken | http://www.linkedin.com/in/nicwise
b. http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop. http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.
http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go: http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding! http://goo.gl/Icp2

Why NOT use memcached? Thats what it’s for :slight_smile: If you run out of space,
it will purge old items, you’ll get a cache miss, and just rerequest
it from the api… (just make sure you test for cache thrashing -
constant misses - and rethink if you are doing this)

You could also (depending on your language) deserialize the results
into objects, and use those objects… basically, build up an in
memory database…

or better yet, use sqlite, which is often an in memory database :slight_smile:
(depending on how you set it up)

What language are you using this in? What context (web app process,
service/daemon, mobile app, etc)?

Serializing it to disk (or just throwing the xml/json on disk) would
work too, but really, its just a larger, more persistant and much,
MUCH slower version of memcached.On Mon, Oct 17, 2011 at 01:12, Ollie me@ollieread.com wrote:

Hai there,

The guys who run the Freeagent twitter account told me to drop on down
here and ask about caching.

I’m working on a system that uses the Freeagent API will be doing a
fair few API calls.

My initial thought was to use memcache but because of the size of the
data I don’t think that’d be appropriate.
My second thought was to save the results as xml files and set an
expiry time within the code for each type of call, and obviously when
the data was updated via my system.

Just wanted to know if you guys have any better thoughts on it, like
methods you’ve used.

Regards
Ollie


You received this message because you are subscribed to the Google Groups “FreeAgent API” group.
To post to this group, send email to freeagent_api@googlegroups.com.
To unsubscribe from this group, send email to freeagent_api+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/freeagent_api?hl=en.

Nic Wise
t. +44 7788 592 806 | @fastchicken | http://www.linkedin.com/in/nicwise
b. http://www.fastchicken.co.nz/

Nearest Bus: find when the next bus is coming to your stop. http://goo.gl/Vcz1p
mobileAgent (for FreeAgent): get your accounts in your pocket.
http://goo.gl/IuBU
Trip Wallet: Keep track of your budget on the go: http://goo.gl/ePhKa
London Bike App: Find the nearest Boris Bike, and get riding! http://goo.gl/Icp2