Analysis: The LGF “Front PageView Effect”

Last week, we exposed an “error” in the way the custom-built LGF page view counter reacted to visitors’ clicks, and touched on a few other causes of page view inflation.  Since then, CJ appears to have corrected the IE problem that I demonstrated in our video (although we’re not convinced that this was the only “bug”), but what remains as the largest culprit to inflated thread page view numbers is the one in plain sight: the “front page effect” (fpe).

First, I’d like to say that we say “plain sight”, because CJ did explain exactly how it works and admitted that it would significantly increase the page view number that is displayed at the top of each thread.  So, while this explanation was buried in the comment section of an unrelated thread, we can’t claim that this trick was snuck in without telling anyone about it.  For the sake of thoroughness, here is CJ’s comment one more time:

Instead, The Boiler Room was naturally curious if there was a way to quantify this effect, and therefore get an idea on the level of bias it adds when comparing page view numbers to all the other websites which don’t employ this technique (and/or when it is used to trash tweet).  Additionally, this kind of data might come in handy if another blogger was thinking about doing something similar. What we found is that this isn’t that hard to do with some sampling and a little statistical analysis. 

For our analysis, here’s what we have to work with:

  • CJ has set 12 threads to display on LGF’s front page (at the time the fpe was announced, it was set at 10).
  • Each front page thread gets a “view” count when the front page is hit.
  • The view counters are observable.
  • Each thread is timestamped to the minute.

We’ve also got some smart and resourceful people here in The Boiler Room, and we can set things up so that the data can be gleaned from automated samples and fed into a database to be charted and graphed.  In short, we can track the reported view increases for any LGF thread from publication until it drops off the front page (and beyond).

What CJ may or may not have realized is that, with just those few things, we can actually get a pretty good idea of levels and patterns in LGF’s front page traffic by simply tracking what happens to these page view counters over time.   Apply a little math and logic, and we can separate the approximate fpe number from the “real” views by applying 2 rules (and these are key, so they deserve bolding):

1. The fpe # can never be greater than the lowest view increase amongst the 12 front page threads over the sample period (except in cases where a new thread is published in between samples and yields the lowest number).  In other words, the increase from the “deadest” thread on the front page contains the highest % of fpe views.

2. The greater the sampling frequency, the more accurate our estimate of the fpe becomes, and the % of fpe views in the increase approaches 100.

For 1, we can’t assume that the lowest view increase # is 100%  front page views, rather that it still may include a few other views that come from click-throughs, referrers, searches, etc., but we know that it will be the closest to the true fpe #.  But based on observation, and knowing generally what happens to views as a thread ages and moves down the front page of a blog, along with the fact that we have 12 threads to sample for the “deadest” and do so frequently, we can say that it’s going to be a very close estimate. 

For 2, we realize that we must balance the effect that our own samples have on the data, as every time we do it we register a front page view ourselves, so we wanted to limit our influence to only 1-2% if possible.  This balance was found taking samples a few times an hour.

So, there you have the methodology.  Take snapshots of the view increase of a thread, and each time subtract the increase of the “deadest” thread on the page, and what you’re left with is the increase that couldn’t have come from fpe (therefore, “real” views).  Make sense?

But, before we reveal the graph and the data, we should ask ourselves: Knowing about this fpe effect, what would we expect the page view counter increases for any given thread on a relatively popular community-style blog like LGF to look like, from the time it’s first published to where it later moves down (and eventually off) the front page? 

A: We’d expect it to increase very rapidly when first published, because in addition to the fpe, you have the lizards and lurkers who will click through to the comments, and the outside referrers (from twitter, other blogs etc.), and refreshing while the thread is “active”.  Then, as the thread ages and moves down the front page, we’d expect the increases to level off slightly, as the extra views from this thread activity dies off and you’re left with mostly fpe views increasing the counter steadily (with “waves”, as time of day will effect front page view rate) until it reaches the bottom of that front page.  Finally, we’d expect the increases to virtually flat-line the minute it is bumped off the front page and becomes thread #13, as it will no longer get fpe increases. 

And what would we expect a non-fpe counter to look like for the same thread? 

A: We should also see a steep increase at first (although not as steep, and not in the same quantities, obviously), and see that taper off as it becomes older and moves down the front page.  After the thread got to be a day old or about 4 spots down on the front page, the thread would essentially be dead for most commenting activity, but we should still see some increases from delayed lurker click throughs, lizards coming back to read comments they missed, searches, etc., and perhaps even a “bump” if/when it sees late hits from other sites.  It’s obviously going to vary a bit by the nature of the thread (for example, we wouldn’t expect an “open thread” to get late traffic from outside referrers, where others may get a lot more; so again, 12 to sample from helps), but for the most part, “real” page view increases should reduce themselves to a creeping pace with periodic bumps by the time the thread is a day old.

Well, we tracked and charted one, so what did we find?

Using a random thread that shall remain anonymous*, from the moment it was hatched to beyond the front page (the #s indicate the changes in its position on the page):

click to enlarge

The red line represents page views recorded from the counter.  Now, remember that with rule #1, the blue and green lines are estimates; it is much more difficult to pin down exactly.  Again, this blue line represents the lowest “real” views could possibly be, and the true line is undoubtedly a little higher for this particular thread (if another thread were sampled, we may see a blue line that is significantly closer to the green). But, since we believe that our methodology is sound, we can say that we’re darn close (to the point where you wouldn’t see much difference in the graph).

Alot of this is fairly intuitive, since the effect stipulates that these dead threads will keep accumulating “views” as long as they’re on the front page.  No one should surprised to see the view counters on these threads to show higher and higher numbers as you scroll down to #12, simply because those threads have been there longer.  So the effect is fairly clear to anyone who stops by LGF and takes a quick glance at all the view counters. 

In conclusion, the point of this exercise was not to prove beyond a doubt that thread 37xxx really got only x number “real” views (as most blogs count them), but to demonstrate the magnitude of the fpe inflation, and show that the technique renders the individual view counters meaningless.  Specifically, the “Front PageView Effect” puts so much weight on the counters that you can’t discern if one thread has a higher count than another because it was particularly insightful/important, or because of thread scheduling it just happened to sit on the front page longer.   That’s why normal blogs have a separate counter for “front page” views, and probably the biggest reason why a claim like this

is rather ridiculous, and deserves to be smacked down.

*the thread # is anonymous for IP security reasons

(Hat tip: The Boiler Room)

Advertisements

Chuck smears again

Today long time poster ryannon was banned from LGF.  Ryannon then came over to the Diary of Daedalus top discuss their banning. No nasty comments were made about Charles “Icarus” Johnson.

Chuck publicly claims he received an email that ryannon was talking bad about him.

Chuck is once again lying. He never received an email about ryannon’s comments on DOD. The Jazz man saw them himself since he reads this blog. Also, ryannon never said anything nasty so he lied in that regard again.

Charles “Icarus” Johnson smears another person. This is why this blog exists, to expose an intellectually dishonest smear merchant.

Deal with it Chuck.

Update: Chuck smeared ryannon even more.

Chuck thinks any discussion about him is talking trash. He really thinks he’s beyond criticism.

Chuck your just a Jazz guitarist, not some shaman.

(Update Hatip Tip: Bagua)


Does @Lizardoid Know His Tweets Are Featured At HuffPo?

There’s something weird going on in the Lizardoid Twitterverse lately, as The Boiler Room has noticed that the “followers” have been increasing at a faster-than-usual rate. We had engineer No. 3 take a closer look, and sure enough, CJ is almost keeping pace with the Shrieking Harpy and R.S. McCain:

And look, he’s almost catching up to Patterico!

And then we saw this sitting in the “chest beat” file, 3 days ago:

That is quite a jump to suddenly go from 1400ish to 1600ish within a few weeks, considering that the account was opened almost 2 years ago. What happened?

Well, we couldn’t find any mention of it in the LGF comments section, but our favorite anonymous DoD commenter brought to our attention that the Lizardoid is having his tweets featured at The Huffington Post!

They show up in that neat scroll bar at the top, and have their own comment section and everything.  I even spotted one of his #nowplaying tweets on there (and as you can see, he must have blocked a few followers; he’s below 1600). 

Congrats CJ! 

But, naturally we wonder if he is even aware of this, ’cause The World’s Greatest Blog Search Engine™ couldn’t find where Charles had announced anything or otherwise mentioned it.  On Google, I had a real hard time finding any of these that were older than a few weeks ago, so the consensus is that it has to be a fairly recent development.  But why haven’t we heard about it?  Wouldn’t ya think that, if he’s gonna brag about the twitter followers,  he’d mention this cool new gig at HuffPo? 

I mean, us LGF old timers remember him talkin’ about that site all the time.   

Well…Okay.  But at least they’re not like a hate site or anything, right?

Oh, forty three forceful updings for that one….absolutely a hate site.

hmmm…

Um, CJ?…about those Twitter followers….

(Hat tip: The Boiler Room, blogwarriorx)


Chuck’s bad prediction

Chuck claims he’s a great political analysis in touch with the public opinion. In December of 2009, he made a prediction about the 2010 midterms. As we see thanks to the efforts of the Boiler Room Crew, it didn’t turn out the way he predicted.

The Result of the November 2nd 2010 midterms a disaster for the Democrats. They lost 6 Senators, 63 House seats, 6 Governorships and scores of state legislator seats. Clearly Charles didn’t admit he was wrong on his prediction. He was silent about the fact he was wrong.

Chuck is no political analyst nor a Nostradamus.  I’ll give him credit about one thing, he didn’t delete this comment!

(Hat Tip: The Boiler Room Crew)

Update: Here is another one of Chuck’s analysis that that is way off.

Chuck clearly doesn’t know what he’s talking about.

(Hat Tip: blogwarriorx)

Update II: On the same thread as Chuck’s bad prediction, another wise and insightful person analyzes.

This was one of Iceweasel’s victims.


This is what all the cool kids will be wearing this summer.

A commenter on an earlier thread opined about a Diary of Daedelus Store. Everyone who opines here gets our attention, and his/her opinion is no different. So let’s look at our options.  Please review these, vote for your favorites, and keep in mind that the winner(s) will likely appear on the apparel.  That means t-shirts and sweatshirts to you 1.0 lurkers.

(Click on any image to see it get really really big fast.)

Vote for your two favorites, and if your nic is Reggie, click on “Other” as many times as you want.  The final design will include “www.diaryofdaedelus.com” at the bottom, presented in a very tasteful and unobtrusive manner.

[Update: This Poll will be open for one week. The DoD Apparel Shop is being set up pending results from the Poll. Anyone choosing “Other” in the Poll is axed to leave a message describing “Other,” and we may or may not have a runoff poll.

Please vote based upon what you yourself would actually purchase, whether it’s a coffee mug or cool clothing, and not solely upon your favorite images. –Briareus]


Chuck sanity doesn’t stop his hypocrisy

Chuck has snapped out of his delusion and finally is thinking clearly. He states that a Muslim Brotherhood takeover of Egypt would be a disaster.

However as you will see, the Muslim Brotherhood has supporters among Chuck’s Far Left readers. Jordash and Talkingpoint Detective are just 2 examples.

As of this writing supporters of the Muslim Brotherhood are not being banned. This shows the hypocrisy of Chuck who bans anyone that expresses support for Conservatives like Glenn Beck. In the new LGF, One can support Islamic Imperialists like the Muslim Brother, but one can’t support The Tea Party.

This is just another example of Chuck’s hypocrisy.

Update: In Chuck’s defense, he has to allow Muslim Brotherhood supporters to post. If he bans them, he will have at most 6 readers left. So I will not judge him on this.

 


Chuck’s Ironic title

Talk about irony, the following title on Chuck’s latest post.

When Chuck tries explaining why his page count jumps, he makes it worse! He should either speak the truth or not be deceitful at all. He never sees the ironic in some of his post titles.