Python Report Generator

Hi all,

I have written a Python application with a GUI that parses and writes the
json response from the API to a CSV file. I have this side working fine
but…

My next hurdle is the paginated results and being able to iterate through
all pages.

Here’s the rather ugly code I have created myself which kind of works but
with very slow and unreliable results:

#URL PAGINATION
pre_number_string = (’?page=’)
page_number = 1
per_page = (’&per_page=100’)
pagination = pre_number_string + str(page_number) + per_page

/// START OF REPORT REQUEST ///

    def getUserData():

        c = pycurl.Curl()
        c.setopt(pycurl.URL, "%s" % str(freeagenturl))
        c.setopt(pycurl.HTTPHEADER, [
            response,
            'Authorization: Bearer '+access_token[3],
            'Accept: application/json',
            'Content-Type: application/json'
            ])
        c.setopt(pycurl.HEADERFUNCTION, headerresponse.write)
        c.setopt(pycurl.WRITEFUNCTION, buf.write)
        c.setopt(pycurl.VERBOSE, True)
        c.setopt(c.SSL_VERIFYPEER, 0)
        c.perform()
    getUserData()

    #HEADER RESPONSE FOR PAGINATION
    header_response = headerresponse.getvalue()
    headersplit = header_response.split(';')
    headercounter = 1

    #LOOP THROUGH PAGES
    if reporttypeholder == 'expenses':
        headerint = int(headersplit[2][57:-14]) #THIS OBTAINS THE TOTAL NUMBER OF PAGES FROM THE PYCURL HEADER
        while headercounter <= headerint:
            page_number = page_number + 1
            pagination = str(page_number) + per_page
            if projectnumberholder == "":
                freeagenturl = "https://api.freeagent.com/v2/" + "%s" % str(reporttypeholder) + pagination + "%s" % str(reportstatusholder)
            else:
                freeagenturl = "https://api.freeagent.com/v2/" + "%s" % str(reporttypeholder) + pagination + "%s" % str(reportstatusholder) + "%s" % str(projectnumberurlprefix) + "%s" % str(projectnumberholder)
            getUserData()
            headercounter = headercounter + 1

    reportresponse = buf.getvalue()
    # /// END OF REPORT REQUEST ///

Does anyone know of a module that will carry this out in a more
Pythonic/cleaner way? Or can anyone show how i can clean my code up? Or a
more official Freeagent solution?

Thanks,
Karl

Hi Karl,

I wrote up a small Python script to request data from the FreeAgent API a
few days ago. It uses the requests module (
http://docs.python-requests.org/en/latest/) which makes the whole HTTP
thing very easy. The script doesn’t include any token retrieval stuff so
you will need to already have an access token to use it. It doesn’t do
pagination (yet) but from what I’ve seen of the requests module it’ll be
much easier than PyCURL seems to make it. Hopefully it might be of some
help.

Here’s the script: https://gist.github.com/Haegin/9943b38cbddd41c7169c

Regards,
Harry MillsOn Thu, Jan 16, 2014 at 9:23 AM, Karl Sayle karlsayle@gmail.com wrote:

Hi all,

I have written a Python application with a GUI that parses and writes the
json response from the API to a CSV file. I have this side working fine
but…

My next hurdle is the paginated results and being able to iterate through
all pages.

Here’s the rather ugly code I have created myself which kind of works but
with very slow and unreliable results:

#URL PAGINATION
pre_number_string = (’?page=’)
page_number = 1
per_page = (’&per_page=100’)
pagination = pre_number_string + str(page_number) + per_page

/// START OF REPORT REQUEST ///

    def getUserData():

        c = pycurl.Curl()
        c.setopt(pycurl.URL, "%s" % str(freeagenturl))
        c.setopt(pycurl.HTTPHEADER, [
            response,
            'Authorization: Bearer '+access_token[3],
            'Accept: application/json',
            'Content-Type: application/json'
            ])
        c.setopt(pycurl.HEADERFUNCTION, headerresponse.write)
        c.setopt(pycurl.WRITEFUNCTION, buf.write)
        c.setopt(pycurl.VERBOSE, True)
        c.setopt(c.SSL_VERIFYPEER, 0)
        c.perform()
    getUserData()

    #HEADER RESPONSE FOR PAGINATION
    header_response = headerresponse.getvalue()
    headersplit = header_response.split(';')
    headercounter = 1

    #LOOP THROUGH PAGES
    if reporttypeholder == 'expenses':
        headerint = int(headersplit[2][57:-14]) #THIS OBTAINS THE TOTAL NUMBER OF PAGES FROM THE PYCURL HEADER
        while headercounter <= headerint:
            page_number = page_number + 1
            pagination = str(page_number) + per_page
            if projectnumberholder == "":
                freeagenturl = "https://api.freeagent.com/v2/" + "%s" % str(reporttypeholder) + pagination + "%s" % str(reportstatusholder)
            else:
                freeagenturl = "https://api.freeagent.com/v2/" + "%s" % str(reporttypeholder) + pagination + "%s" % str(reportstatusholder) + "%s" % str(projectnumberurlprefix) + "%s" % str(projectnumberholder)
            getUserData()
            headercounter = headercounter + 1

    reportresponse = buf.getvalue()
    # /// END OF REPORT REQUEST ///

Does anyone know of a module that will carry this out in a more
Pythonic/cleaner way? Or can anyone show how i can clean my code up? Or a
more official Freeagent solution?

Thanks,
Karl


You received this message because you are subscribed to the Google Groups
“FreeAgent API” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to freeagent_api+unsubscribe@googlegroups.com.
To post to this group, send email to freeagent_api@googlegroups.com.
Visit this group at http://groups.google.com/group/freeagent_api.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Harry,

Nice looking script. I did look at the requests module originally but
personally found Pycurl easier to read/work with. I have got all that side
working perfectly as well as the refresh token. The problem is, like
yourself, working around the pagination to ‘catch all’ the lines from the
API.

The loop I created to attempt to re-query the API does work and you can
watch it changing the page number but the script either crashes or gives a
huge, virtually unusable variable:

reportresponse = buf.getvalue()

Without using pagination this works really well including the GUI.

If you’d like to see my script for the token refresh please let me know.

Thanks again for your help,
KarlOn Thursday, January 16, 2014 9:57:47 AM UTC, Harry wrote:

Hi Karl,

I wrote up a small Python script to request data from the FreeAgent API a
few days ago. It uses the requests module (
http://docs.python-requests.org/en/latest/) which makes the whole HTTP
thing very easy. The script doesn’t include any token retrieval stuff so
you will need to already have an access token to use it. It doesn’t do
pagination (yet) but from what I’ve seen of the requests module it’ll be
much easier than PyCURL seems to make it. Hopefully it might be of some
help.

Here’s the script: https://gist.github.com/Haegin/9943b38cbddd41c7169c


Regards,
Harry Mills

On Thu, Jan 16, 2014 at 9:23 AM, Karl Sayle <karl...@gmail.com<javascript:> wrote:

Hi all,

I have written a Python application with a GUI that parses and writes the
json response from the API to a CSV file. I have this side working fine
but…

My next hurdle is the paginated results and being able to iterate through
all pages.

Here’s the rather ugly code I have created myself which kind of works but
with very slow and unreliable results:

#URL PAGINATION
pre_number_string = (’?page=’)
page_number = 1
per_page = (’&per_page=100’)
pagination = pre_number_string + str(page_number) + per_page

/// START OF REPORT REQUEST ///

    def getUserData():

        c = pycurl.Curl()
        c.setopt(pycurl.URL, "%s" % str(freeagenturl))
        c.setopt(pycurl.HTTPHEADER, [
            response,
            'Authorization: Bearer '+access_token[3],
            'Accept: application/json',
            'Content-Type: application/json'
            ])
        c.setopt(pycurl.HEADERFUNCTION, headerresponse.write)
        c.setopt(pycurl.WRITEFUNCTION, buf.write)
        c.setopt(pycurl.VERBOSE, True)
        c.setopt(c.SSL_VERIFYPEER, 0)
        c.perform()
    getUserData()

    #HEADER RESPONSE FOR PAGINATION
    header_response = headerresponse.getvalue()
    headersplit = header_response.split(';')
    headercounter = 1

    #LOOP THROUGH PAGES
    if reporttypeholder == 'expenses':
        headerint = int(headersplit[2][57:-14]) #THIS OBTAINS THE TOTAL NUMBER OF PAGES FROM THE PYCURL HEADER
        while headercounter <= headerint:
            page_number = page_number + 1
            pagination = str(page_number) + per_page
            if projectnumberholder == "":
                freeagenturl = "https://api.freeagent.com/v2/" + "%s" % str(reporttypeholder) + pagination + "%s" % str(reportstatusholder)

            else:
                freeagenturl = "https://api.freeagent.com/v2/" + "%s" % str(reporttypeholder) + pagination + "%s" % str(reportstatusholder) + "%s" % str(projectnumberurlprefix) + "%s" % str(projectnumberholder)

            getUserData()
            headercounter = headercounter + 1

    reportresponse = buf.getvalue()
    # /// END OF REPORT REQUEST ///

Does anyone know of a module that will carry this out in a more
Pythonic/cleaner way? Or can anyone show how i can clean my code up? Or a
more official Freeagent solution?

Thanks,
Karl


You received this message because you are subscribed to the Google Groups
“FreeAgent API” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to freeagent_ap...@googlegroups.com <javascript:>.
To post to this group, send email to freeag...@googlegroups.com<javascript:>
.
Visit this group at http://groups.google.com/group/freeagent_api.
For more options, visit https://groups.google.com/groups/opt_out.