Slow as a snake for heavy blog

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow as a snake for heavy blog

Emmanuel Bernard
Hey guys,

We are trying to migrate http://in.relation.to to Awestruct.
That's a bog with ~1000 entries, ~100 tags and ~20 authors.

We ahve done our initial set up and imports but the build time is
horrific.

I did get the generation go through entirely with all the posts with
memory topping at 329MB. It took 28 minutes. However the first page rendered crashed the
process.
Restarting awestruct got me through but even with all the pages cached,
it takes close to 3 minutes to start and the first rendered page crashes
ruby.

However rendering pages from _site with
some other web server works fine and the site seems perfect.
So all pages are indeed rendered.

The time is quite prohibitive. I tried installing
awestruct for JRuby but that did not work.

What is killing us is that process I suspect is the exponential
combination between posts, tags and authors as we do build pagination
and atom feed for each author and tag:

- find all posts and put them in a list
- walk through the list and create the paging pages
- walk through the list and create the list of tags and stock the list
  of pages for a tag; for each author create the tag pages and their corresponding
  paging pages
- walk through the list and create the list of authors and stock the list
  of pages for an author ; for each author create the author pages and their corresponding
  paging pages
- walk through all posts and create the global feed
- walk through all posts per tag and create a feed per tag
- walk through all posts per author and create a feed per author

It seems that each feed (except the global one) and each paginated page
takes around 1s. With the shear number of posts + tags we have, it

It's sad, all of the other pieces were falling in place perfectly :(

Do you guys have any help or ideas for me to reduce that to a bearable
time?

Emmanuel


---------------------------------------------------------------------
Archives: http://talk-archive.awestruct.org/
To unsubscribe, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Slow as a snake for heavy blog

bobmcwhirter
Administrator

For the walk-all-posts-and-DO_SOMETHING stuff, maybe combine them into a single extension, that walks all posts once, building N collections during the single walk?

Likewise, during the same single walk, build up collections for tags'n'such?

        -Bob


On Sep 28, 2012, at 11:04 AM, Emmanuel Bernard wrote:

> Hey guys,
>
> We are trying to migrate http://in.relation.to to Awestruct.
> That's a bog with ~1000 entries, ~100 tags and ~20 authors.
>
> We ahve done our initial set up and imports but the build time is
> horrific.
>
> I did get the generation go through entirely with all the posts with
> memory topping at 329MB. It took 28 minutes. However the first page rendered crashed the
> process.
> Restarting awestruct got me through but even with all the pages cached,
> it takes close to 3 minutes to start and the first rendered page crashes
> ruby.
>
> However rendering pages from _site with
> some other web server works fine and the site seems perfect.
> So all pages are indeed rendered.
>
> The time is quite prohibitive. I tried installing
> awestruct for JRuby but that did not work.
>
> What is killing us is that process I suspect is the exponential
> combination between posts, tags and authors as we do build pagination
> and atom feed for each author and tag:
>
> - find all posts and put them in a list
> - walk through the list and create the paging pages
> - walk through the list and create the list of tags and stock the list
>  of pages for a tag; for each author create the tag pages and their corresponding
>  paging pages
> - walk through the list and create the list of authors and stock the list
>  of pages for an author ; for each author create the author pages and their corresponding
>  paging pages
> - walk through all posts and create the global feed
> - walk through all posts per tag and create a feed per tag
> - walk through all posts per author and create a feed per author
>
> It seems that each feed (except the global one) and each paginated page
> takes around 1s. With the shear number of posts + tags we have, it
>
> It's sad, all of the other pieces were falling in place perfectly :(
>
> Do you guys have any help or ideas for me to reduce that to a bearable
> time?
>
> Emmanuel
>
>
> ---------------------------------------------------------------------
> Archives: http://talk-archive.awestruct.org/
> To unsubscribe, e-mail: [hidden email]
>


---------------------------------------------------------------------
Archives: http://talk-archive.awestruct.org/
To unsubscribe, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Slow as a snake for heavy blog

LightGuard
Administrator
I would agree, the more you can do in one extension the easier and faster that's going to become. You could look into some threading stuff and try something like a fork/join approach, but that may be overkill and probably wouldn't help a whole lot unless you're on jruby.

On that note, I've often wondered if extensions could / should be called concurrently to improve generation perf, or perhaps a user defined set of extensions. More often than not extensions are self contained.

On Fri, Sep 28, 2012 at 9:20 AM, Bob McWhirter <[hidden email]> wrote:

For the walk-all-posts-and-DO_SOMETHING stuff, maybe combine them into a single extension, that walks all posts once, building N collections during the single walk?

Likewise, during the same single walk, build up collections for tags'n'such?

        -Bob


On Sep 28, 2012, at 11:04 AM, Emmanuel Bernard wrote:

> Hey guys,
>
> We are trying to migrate http://in.relation.to to Awestruct.
> That's a bog with ~1000 entries, ~100 tags and ~20 authors.
>
> We ahve done our initial set up and imports but the build time is
> horrific.
>
> I did get the generation go through entirely with all the posts with
> memory topping at 329MB. It took 28 minutes. However the first page rendered crashed the
> process.
> Restarting awestruct got me through but even with all the pages cached,
> it takes close to 3 minutes to start and the first rendered page crashes
> ruby.
>
> However rendering pages from _site with
> some other web server works fine and the site seems perfect.
> So all pages are indeed rendered.
>
> The time is quite prohibitive. I tried installing
> awestruct for JRuby but that did not work.
>
> What is killing us is that process I suspect is the exponential
> combination between posts, tags and authors as we do build pagination
> and atom feed for each author and tag:
>
> - find all posts and put them in a list
> - walk through the list and create the paging pages
> - walk through the list and create the list of tags and stock the list
>  of pages for a tag; for each author create the tag pages and their corresponding
>  paging pages
> - walk through the list and create the list of authors and stock the list
>  of pages for an author ; for each author create the author pages and their corresponding
>  paging pages
> - walk through all posts and create the global feed
> - walk through all posts per tag and create a feed per tag
> - walk through all posts per author and create a feed per author
>
> It seems that each feed (except the global one) and each paginated page
> takes around 1s. With the shear number of posts + tags we have, it
>
> It's sad, all of the other pieces were falling in place perfectly :(
>
> Do you guys have any help or ideas for me to reduce that to a bearable
> time?
>
> Emmanuel
>
>
> ---------------------------------------------------------------------
> Archives: http://talk-archive.awestruct.org/
> To unsubscribe, e-mail: [hidden email]
>


---------------------------------------------------------------------
Archives: http://talk-archive.awestruct.org/
To unsubscribe, e-mail: [hidden email]




--
Jason Porter
http://lightguard-jp.blogspot.com
http://twitter.com/lightguardjp

Software Engineer
Open Source Advocate
Author of Seam Catch - Next Generation Java Exception Handling

PGP key id: 926CCFF5
PGP key available at: keyserver.net, pgp.mit.edu
Reply | Threaded
Open this post in threaded view
|

Re: Slow as a snake for heavy blog

cody.lerum
Seems logical that Paginator, Tagger, TagCloud, Atomizer, etc would
all fall under a single Posts extension.

-C

On Fri, Sep 28, 2012 at 9:24 AM, Jason Porter <[hidden email]> wrote:

> I would agree, the more you can do in one extension the easier and faster
> that's going to become. You could look into some threading stuff and try
> something like a fork/join approach, but that may be overkill and probably
> wouldn't help a whole lot unless you're on jruby.
>
> On that note, I've often wondered if extensions could / should be called
> concurrently to improve generation perf, or perhaps a user defined set of
> extensions. More often than not extensions are self contained.
>
>
> On Fri, Sep 28, 2012 at 9:20 AM, Bob McWhirter <[hidden email]> wrote:
>>
>>
>> For the walk-all-posts-and-DO_SOMETHING stuff, maybe combine them into a
>> single extension, that walks all posts once, building N collections during
>> the single walk?
>>
>> Likewise, during the same single walk, build up collections for
>> tags'n'such?
>>
>>         -Bob
>>
>>
>> On Sep 28, 2012, at 11:04 AM, Emmanuel Bernard wrote:
>>
>> > Hey guys,
>> >
>> > We are trying to migrate http://in.relation.to to Awestruct.
>> > That's a bog with ~1000 entries, ~100 tags and ~20 authors.
>> >
>> > We ahve done our initial set up and imports but the build time is
>> > horrific.
>> >
>> > I did get the generation go through entirely with all the posts with
>> > memory topping at 329MB. It took 28 minutes. However the first page
>> > rendered crashed the
>> > process.
>> > Restarting awestruct got me through but even with all the pages cached,
>> > it takes close to 3 minutes to start and the first rendered page crashes
>> > ruby.
>> >
>> > However rendering pages from _site with
>> > some other web server works fine and the site seems perfect.
>> > So all pages are indeed rendered.
>> >
>> > The time is quite prohibitive. I tried installing
>> > awestruct for JRuby but that did not work.
>> >
>> > What is killing us is that process I suspect is the exponential
>> > combination between posts, tags and authors as we do build pagination
>> > and atom feed for each author and tag:
>> >
>> > - find all posts and put them in a list
>> > - walk through the list and create the paging pages
>> > - walk through the list and create the list of tags and stock the list
>> >  of pages for a tag; for each author create the tag pages and their
>> > corresponding
>> >  paging pages
>> > - walk through the list and create the list of authors and stock the
>> > list
>> >  of pages for an author ; for each author create the author pages and
>> > their corresponding
>> >  paging pages
>> > - walk through all posts and create the global feed
>> > - walk through all posts per tag and create a feed per tag
>> > - walk through all posts per author and create a feed per author
>> >
>> > It seems that each feed (except the global one) and each paginated page
>> > takes around 1s. With the shear number of posts + tags we have, it
>> >
>> > It's sad, all of the other pieces were falling in place perfectly :(
>> >
>> > Do you guys have any help or ideas for me to reduce that to a bearable
>> > time?
>> >
>> > Emmanuel
>> >
>> >
>> > ---------------------------------------------------------------------
>> > Archives: http://talk-archive.awestruct.org/
>> > To unsubscribe, e-mail: [hidden email]
>> >
>>
>>
>> ---------------------------------------------------------------------
>> Archives: http://talk-archive.awestruct.org/
>> To unsubscribe, e-mail: [hidden email]
>>
>
>
>
> --
> Jason Porter
> http://lightguard-jp.blogspot.com
> http://twitter.com/lightguardjp
>
> Software Engineer
> Open Source Advocate
> Author of Seam Catch - Next Generation Java Exception Handling
>
> PGP key id: 926CCFF5
> PGP key available at: keyserver.net, pgp.mit.edu

---------------------------------------------------------------------
Archives: http://talk-archive.awestruct.org/
To unsubscribe, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Slow as a snake for heavy blog

Emmanuel Bernard
In reply to this post by bobmcwhirter
That was what I had in mind as an experiment. I'll keep you posted.

On Fri 2012-09-28 11:20, Bob McWhirter wrote:

>
> For the walk-all-posts-and-DO_SOMETHING stuff, maybe combine them into a single extension, that walks all posts once, building N collections during the single walk?
>
> Likewise, during the same single walk, build up collections for tags'n'such?
>
> -Bob
>
>
> On Sep 28, 2012, at 11:04 AM, Emmanuel Bernard wrote:
>
> > Hey guys,
> >
> > We are trying to migrate http://in.relation.to to Awestruct.
> > That's a bog with ~1000 entries, ~100 tags and ~20 authors.
> >
> > We ahve done our initial set up and imports but the build time is
> > horrific.
> >
> > I did get the generation go through entirely with all the posts with
> > memory topping at 329MB. It took 28 minutes. However the first page rendered crashed the
> > process.
> > Restarting awestruct got me through but even with all the pages cached,
> > it takes close to 3 minutes to start and the first rendered page crashes
> > ruby.
> >
> > However rendering pages from _site with
> > some other web server works fine and the site seems perfect.
> > So all pages are indeed rendered.
> >
> > The time is quite prohibitive. I tried installing
> > awestruct for JRuby but that did not work.
> >
> > What is killing us is that process I suspect is the exponential
> > combination between posts, tags and authors as we do build pagination
> > and atom feed for each author and tag:
> >
> > - find all posts and put them in a list
> > - walk through the list and create the paging pages
> > - walk through the list and create the list of tags and stock the list
> >  of pages for a tag; for each author create the tag pages and their corresponding
> >  paging pages
> > - walk through the list and create the list of authors and stock the list
> >  of pages for an author ; for each author create the author pages and their corresponding
> >  paging pages
> > - walk through all posts and create the global feed
> > - walk through all posts per tag and create a feed per tag
> > - walk through all posts per author and create a feed per author
> >
> > It seems that each feed (except the global one) and each paginated page
> > takes around 1s. With the shear number of posts + tags we have, it
> >
> > It's sad, all of the other pieces were falling in place perfectly :(
> >
> > Do you guys have any help or ideas for me to reduce that to a bearable
> > time?
> >
> > Emmanuel
> >
> >
> > ---------------------------------------------------------------------
> > Archives: http://talk-archive.awestruct.org/
> > To unsubscribe, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> Archives: http://talk-archive.awestruct.org/
> To unsubscribe, e-mail: [hidden email]
>

---------------------------------------------------------------------
Archives: http://talk-archive.awestruct.org/
To unsubscribe, e-mail: [hidden email]