31 replies to this topic

[joneill809]

Hello! This is my first post - I couldn't figure out where it belongs, so my apologies if it shouldn't be in general nerf. I'm just getting into the hobby in the past few months, and I was lucky enough to pickup one of Captain Slug's Caliburn kits. I recently completed the blaster and ran it through the chronograph today. Honestly I'm not into going to wars, but rather the statistics around different blaster configurations really fascinate me (in addition to pew-pewing cups with my kids around the house / backyard). I enjoy data collection and visualization, so I started thinking of some interesting studies that would generate data sets that would make for some cool plots. I've amassed a relatively comprehensive set of readings from a variety of blaster configurations focused mostly on Retaliators (and I figured it would be cool to throw the Caliburn into the mix), which raised the question - how does everyone compare chronograph readings when we don't have a method for baselining / comparing standardized results? 

 

Here's what I have for a series of comparisons of 100 shots with each blaster:

 

 

Each plot represents the population of readings in light gray dots. The minimum and maximum readings for the population are reported as horizontal bars and labeled "max" and "min". The average for the population is the large horizontal bar labeled with the largest font. The white circle is the median value. The two smaller horizontal bars are the 25th and 75th percentile values. The "+/-" values reported as part of the FPS value are the 95th percentile confidence intervals for the population, meaning you would expect 95 of 100 shots to fall within that range (~2 standard deviations) of the average. I can provide the full data sets for those interested.

 

I'm really impressed with the consistency of the Caliburn. Note how awesome and symmetrical that population plot looks - where the min, 25th, 50th, average, 75th and max fall where we'd expect. You see more variation in the results from my retaliator studies (these were all within 1 foot of the chronograph). 

 

So on to my problem.

 

My chronograph, a Caldwell Ballistic Precision Chronograph, *seems* to record fps readings consistently lower than what is typically reported in the hobby. My baseline figures from a stock retaliator come in at about 54 fps, 15 to 20 fps lower than what seem to be commonly accepted hobby standards. I had a feeling my results were consistently low for the Retaliator spring / kit configurations I was testing. This suspicion seems to be playing out with my Caliburn numbers - my chronograph is reporting ~172 fps, is ~20% lower than what I expected. This combined with the ~25% lower readings on my baseline Retaliator makes me suspect my chronograph is reporting low figures.

 

I picked up the Caldwell at a good sale price, but I'm thinking I need a different chronograph. Has anyone else seen consistently low results from Caldwell's? I see quite a few other folks using Competition Electronics units - I'm guessing there's a reason  From what I've read these units are factory calibrated, but it doesn't seem like there is a field calibration method. I fully admit I'm new to this, so I may have missed something in my searching. Hence my post.

 

Anyhow given how accurate some of these upgraded blasters are I've been looking into conducting FPS and accuracy studies at a variety of ranges for a blaster. I've done one analysis on a Retaliator, but before I dig in and produce these studies on more configurations I want to get a chronograph setup that I can trust. Right now, my results are OK for comparative testing on my unit, but I'd like to have more confidence in the results I'm producing when compared to other folks. This is the type of study I had in mind:

 

 

I'm measuring accuracy as a function of how many readings I can get from the chronograph at a specific distance. In this plot you see the population of readings from the chronograph and the associated statistics in the upper portion of the plot for each distance (x-axis). Below the FPS plots is a figure of reading count. You can see the number of readings drop off as you move further to the right which is a greater distance from the chronograph. The data collection on these is fairly intensive (100 shots from each distance) so I want to have confidence in the chronograph before proceeding with a lot more tests.

 

I'd appreciate any tips anyone has on chronographs, as well as recommended studies where there may be a need within the hobby. Personally the accuracy at various range studies seem intriguing, which is where that second plot is going. Given how accurate some of these blasters are (were I can get a significant population at a decent distance), using this "how many readings do I get at a distance?" seemed like a good start.

 

Thanks in advance!

 

 

[Daniel Beaver]

I'll respond tomorrow when I'm back at my computer. But my gut instinct is that your chronograph results are accurate.

[shandsgator8]

From thermocouples to blood pressure cuffs to blood sugar testers, I know from personal experience and reading spec sheets that getting inaccurate and/or inconsistent results in to be expected from a consumer product and that's what your chronograph appears to be, especially given its price. I know growing up, I wondered why lab equipment was so expensive compared to what you can get from the average store. Turns out it was the equipment's level of accuracy and precision. That's a major reason why professional or mission critical equipment costs significantly more than the "everyday" version of the equipment.

 

As for whether or not your chronograph is inaccurate, I can't say, but it wouldn't surprise me one bit if it's reading lower readings than it should. I guess the real questions are whether you can re-calibrate it and if the inaccuracy is consistent, i.e. you're still able to maintain precision.

Edited by shandsgator8, 18 June 2017 - 07:08 PM.

[Bubba Longshot]

I'm not an expert on chronographs (I don't even own one), but normally people measure their shots 6inches away; not your 12in. Coop, Drac, and Orangemodworks chrono at 6 inches.

Edited by Bubba Longshot, 18 June 2017 - 07:05 PM.

[Meaker VI]

Darts also play a significant role in chrono readings. An elite may mass more than a koosh but less than a waffle, for example. Which is ALSO something you could (& should) test - especially the accuracy between types, as this is a fairly under supported area in nerf research.

IMO, your actual results don't matter as long as YOUR setup is consistent- if you go on to present (beautiful !) statistics like these for many categories of blaster and mod, the actual result can be manipulated later if your chrono is found to be ~10% slow (or whatever). It is only an issue if you are changing the setup between blasters (different lighting, angle, w/e) and/or your results are varying inconsistently ( between shots or blasters).

Speaking of, how is your setup? That's an important part of statistics taking too. Do you fire from a bench, aim each one, ...? Do you have good, consistent lighting for the chrono? Are your darts fresh elites and all one color? Etc. etc.

I would absolutely love to see more, especially a set of stock/modded Stryfes. Then look into the SCAR barrel - I'd love to see results on that. The Hammershot would also probably warrant an extensive test (stock, spacered, 3d printed 7-cylinder, new spring). Obviously the more blasters with the more darts and setups, the better, but to keep a reasonable lid on it I'd leave it with those four to start (Retal, Caliburn, Stryfe, HS).

[FFNerfmodding]

I think your data looks pretty good in comparison to some other spring data I've seen. I would only really be able to tell you about flywheel data, because that's my specialization. Compliments to the way you layed it out too.

Edited by FFNerfmodding, 19 June 2017 - 12:01 AM.

[joneill809]

Daniel Beaver, on 18 Jun 2017 - 11:53 PM, said:

I'll respond tomorrow when I'm back at my computer. But my gut instinct is that your chronograph results are accurate.

 

Thanks! I really appreciate the feedback and guidance.

 

 

shandsgator8, on 19 Jun 2017 - 12:04 AM, said:

From thermocouples to blood pressure cuffs to blood sugar testers, I know from personal experience and reading spec sheets that getting inaccurate and/or inconsistent results in to be expected from a consumer product and that's what your chronograph appears to be, especially given its price. I know growing up, I wondered why lab equipment was so expensive compared to what you can get from the average store. Turns out it was the equipment's level of accuracy and precision. That's a major reason why professional or mission critical equipment costs significantly more than the "everyday" version of the equipment.

 

As for whether or not your chronograph is inaccurate, I can't say, but it wouldn't surprise me one bit if it's reading lower readings than it should. I guess the real questions are whether you can re-calibrate it and if the inaccuracy is consistent, i.e. you're still able to maintain precision.

 

Agree - back when I worked in a chemistry lab we spent a lot of time (and money) on purchasing reference materials and baselining our equipment to profile our accuracy against known references. I was trying to think of a way to apply that to blasters, e.g. find a fairly consistent platform and potentially modify it with a fairly consistent spring, then use that as a reference platform. If my readings are "x" fps lower than the reference then we know where my setup stands. I can also use that approach to monitor for drift over time or under various conditions. 

 

It's hard to imagine getting precise numbers at these low fps ranges when the manufacturer is marketing to the firearms market where the fps ranges are orders of magnitude higher. I thought I might have a chance with this one because they also market to archers, but you never know until you give it a go. I don't think these cheapo / entry-level devices have a field calibration method, or at least one that I have found. I'll hit up the manufacturer again with another round of questions. 

 

 

Bubba Longshot, on 19 Jun 2017 - 12:04 AM, said:

I'm not an expert on chronographs (I don't even own one), but normally people measure their shots 6inches away; not your 12in. Coop, Drac, and Orangemodworks chrono at 6 inches.

 

Sorry about overlooking that detail. I lumped all my muzzle readings into a category of "under 1 foot" because of the different designs of the blasters. I have readings from my two modified Retaliators that are using aftermarket stefan kits from Artifact and Worker. Their barrels (about 4") are nested inside the outer barrel of the pump grip on my Retaliators, so technically the muzzle is 6" further back from the front of the blaster. I don't know how significant this is, and I didn't feel like switching back to the stock top prime with some sort of bolt grip. That 10kg Aussie spring is tough to prime haha. Anyhow I just label them as "1 ft" when capturing the data on my phone and it carried forward to my laptop when I was plotting. 

 

I'm still not sure of the best way to characterize this measurement. Advice here is appreciated!

 

 

Meaker VI, on 19 Jun 2017 - 04:10 AM, said:

Darts also play a significant role in chrono readings. An elite may mass more than a koosh but less than a waffle, for example. Which is ALSO something you could (& should) test - especially the accuracy between types, as this is a fairly under supported area in nerf research.

IMO, your actual results don't matter as long as YOUR setup is consistent- if you go on to present (beautiful !) statistics like these for many categories of blaster and mod, the actual result can be manipulated later if your chrono is found to be ~10% slow (or whatever). It is only an issue if you are changing the setup between blasters (different lighting, angle, w/e) and/or your results are varying inconsistently ( between shots or blasters).

Speaking of, how is your setup? That's an important part of statistics taking too. Do you fire from a bench, aim each one, ...? Do you have good, consistent lighting for the chrono? Are your darts fresh elites and all one color? Etc. etc.

I would absolutely love to see more, especially a set of stock/modded Stryfes. Then look into the SCAR barrel - I'd love to see results on that. The Hammershot would also probably warrant an extensive test (stock, spacered, 3d printed 7-cylinder, new spring). Obviously the more blasters with the more darts and setups, the better, but to keep a reasonable lid on it I'd leave it with those four to start (Retal, Caliburn, Stryfe, HS).

 

Thanks for the comment on "(beautiful !) statistics"! My day job focuses on condensing millions to billions of data points into manageable visualizations, so I enjoy applying that to my hobbies (I'm also into dart frogs where I have a large source of temperature and humidity data). The nerf hobby caught my attention with fps / accuracy / precision data, and I thought this would be an interesting challenge, especially with the variety of variables you can introduce with different mods.

 

I forgot to mention the darts. I baselined the stock Retaliator with streamlines and ekind waffles. Here's a quick screen grab of the summary data from two runs where the variable was dart type:

 

 

This data indicated the two dart types yielded the same FPS results. I can repeat them with a different blaster, but they were pretty consistent. I look mainly at the 95th pct confidence interval of the population because that gives me a better idea of what to expect from any given shot from a blaster. This table has all the base stats I use to make those plots, the population of results is below this summary in my worksheet and I can clean all that up and share it if anyone is interested in the raw data too.

 

I'll admit my setup isn't super consistent yet. I'm hand firing the blasters through the Caldwell on a bar-height table. I did pickup the indoor lighting kit to improve the consistency of the readings from the chronograph. I shoot indoors to keep the temp / RH consistent (we set the target RH for the house and it's really consistent, especially in the summer). I'm in a 15 foot room so I can get my 1, 6 and 12 foot ranges. 18 and 24 are fired through a double door sized opening (best I can do). Using this setup I can extend the ranges to ~50 ft across the house if I get an accurate enough blaster to hit the chronograph from that distance. There's a sterilite bin behind the Caldwell with a towel taped to it that catches darts. I've been thinking of some kind of cart top mount for the worker p-mag / hex-mags that may get a consistent firing angle, but I thought I would get a set of range tests manually before taking that step. If anyone has additional ideas I'm open to feedback!

 

In terms of darts, most of these tests are focused on stefans / short darts. I've tested worker, artifact FVJ's, and ACC Gen 2 soft and hard tips. I mostly test with the hard tips now. I have not tried to make my own [yet]. I have batches of 100 that I'll cycle through for a test, loaded in a variety of worker stefan mags that I have on hand. I have not tried stock darts in the Caliburn yet.

 

One of the tests I mentioned above, reference testing, is something I need to circle back to. One of the concepts I've used in the lab was to periodically retest a configuration and compare the results that I got last week, 2 weeks ago, etc to observe the consistency of the setup over time. In the chem lab we were monitoring for "machine drift" but here it's more likely operator drift or a busted blaster (e.g. I had an o-ring work itself loose in one of the Artifact stefan tests and you could see the impact in the course of the 100 shot test through a step down in the fps results).  I'll get back to re-testing a configuration with one of the modded Retaliators (my original "stock Ret" is now torn apart / dremeled out for an expanded plunger tube mod). 

 

I agree I'm after nailing down the consistency of my setup, and I can test for that through a series of reproducibility tests. That said, given how many blasters are out there, I'd also like to see how we can share results in the community and try to account for variability from setup to setup through some kind of standardization process. Maybe I'm complicating this too much, but that idea of coming up with a blaster that is widely available and consistent, or consistent with some specific mods, could help us all adjust our sharing methodologies. Don't get me wrong, I'd love to test a bunch of different blasters, but I think a deeper dive into coming up with a collection methodology and standardized analysis would let us all profile a lot more blasters. It would be cool to pull together an online repository of results as well (if this doesn't already exist). 

 

I haven't gotten into flywheels [much] yet. My kiddos are young and I enjoy modding blasters they can use so I focused on springers since I'm leery about having them playing with lipos right now. We do have a couple of stryfe's that I'll get to eventually, but I kind of got hooked on springers. And man is the Caliburn fun lol!

 

I too am intrigued by the SCAR barrel. I have been messing with a design that would slip on to the Caliburn. I am starting without a twist. I've read the improvement in accuracy may be from a stabilizing effect rather than the twist so I added 4 parallel ridges to this design and some exhaust holes (it looked cool but maybe they are not needed - I can remove them in another design). I thought I would iterate on this for a while then add a twist later.

 

Anyhow this idea was the main driver behind coming up with a way to baseline accuracy as a function of distance. Here's what I started with.

 

Front orientation (you can see the teeth):

 

Rear orientation that slips on the barrel:

 

My first print was too narrow for the OD of the Caliburn barrel so I need to tune the design a bit more. I also wanted everyone's feedback on this thread before I did a distance profiling on the Caliburn since that work is pretty time consuming (500 shots or more!). I need to do more reading on the SCAR barrel and design a set of experiments around different attachment designs that covers the key variables that surfaced on many of the forum discussions. Feedback here is also more than welcome. Many of these experiments will be long running.

 

FFNerfmodding, on 19 Jun 2017 - 05:01 AM, said:

I think your data looks pretty good in comparison to some other spring data I've seen. I would only really be able to tell you about flywheel data, because that's my specialization. Compliments to the way you layed it out too.

 

Thanks I'm glad the feedback so far seems to be the numbers are coming in OK. My main concern was the delta I was seeing with the reported numbers from folks like Coop and Drac on stock/modded blasters, and Captain Slug's Calibrun numbers. I seem consistently lower, which is what got me thinking on calibration, cross-comparison, and possibly reference configurations. And thanks for the compliment on layout! 

 

 

Apologies for the long post! I squirreled away a lot of data over the past few months that I'm trying to organize. It took a few months to compile nearly two thousand shots of data behind these plots and others I haven't published yet. I figured it was time to gather my thoughts and some feedback before really getting into deep profiling of specific configurations. 

 

I really appreciate all the feedback and help! --Jim

[Meaker VI]

joneill809, on 19 Jun 2017 - 4:22 PM, said:

Here's a quick screen grab of the summary data from two runs where the variable was dart type: ...
 
This data indicated the two dart types yielded the same FPS results. ...

Good! I hear Koosh run hot, so some of your lower numbers might come from that.
 

joneill809, on 19 Jun 2017 - 4:22 PM, said:

I'll admit my setup isn't super consistent yet. I'm hand firing the blasters through the Caldwell on a bar-height table. I did pickup the indoor lighting kit to improve the consistency of the readings from the chronograph. I shoot indoors to keep the temp / RH consistent (we set the target RH for the house and it's really consistent, especially in the summer). I'm in a 15 foot room so I can get my 1, 6 and 12 foot ranges. 18 and 24 are fired through a double door sized opening (best I can do). ... There's a sterilite bin behind the Caldwell with a towel taped to it that catches darts. ...but I thought I would get a set of range tests manually before taking that step. If anyone has additional ideas I'm open to feedback!

Frankly, this setup sounds better thought out than most. If you're after standardization, letting us know your temp, humidity, and altitude is probably important. I think that a rest is probably important for real accuracy testing, maybe one of those portable clamping work tables would work? It may not be though, since at 24' I'd expect most people to be able to aim well enough to hit the chrono barring dart swerve.
 

joneill809, on 19 Jun 2017 - 4:22 PM, said:

In terms of darts, most of these tests are focused on stefans / short darts. I've tested worker, artifact FVJ's, and ACC Gen 2 soft and hard tips. I mostly test with the hard tips now. I have not tried to make my own [yet]. I have batches of 100 that I'll cycle through for a test, loaded in a variety of worker stefan mags that I have on hand. I have not tried stock darts in the Caliburn yet.

Ah, this is where you deviate from the majority. Most players use full-length stock/3rd party darts, so testing those in setups that permit it is nessecary IMO.
 

joneill809, on 19 Jun 2017 - 4:22 PM, said:

One of the tests I mentioned above, reference testing, is something I need to circle back to. One of the concepts I've used in the lab was to periodically retest a configuration and compare the results that I got last week...

Ok - I see how that'd eliminate machine/operator error in data collection. Again, I think you're probably doing it *better* than most, hence your results are more realistic (not all chrono posts go through 100 darts and take the average!).
 

joneill809, on 19 Jun 2017 - 4:22 PM, said:

That said, given how many blasters are out there, I'd also like to see how we can share results in the community and try to account for variability from setup to setup through some kind of standardization process. ...Don't get me wrong, I'd love to test a bunch of different blasters, but I think a deeper dive into coming up with a collection methodology and standardized analysis would let us all profile a lot more blasters. It would be cool to pull together an online repository of results as well (if this doesn't already exist). 

I'm concerned that the trick would be *actually getting people to really do the setup*. I don't think most nerfers would adhere to any standard created, and the false FPS data will continue to circulate. Encourage proper methods anyway, but post new data.

You wouldn't need to test *every* blaster with *every* configuration either - some blasters either aren't worth testing or are the same as other blasters, some setups aren't worth testing (unavailable) or are similar to other setups, and people doing unique stuff are either doing it for their own sake or to push the envelope. People pushing the envelope usually are the ones who have the means to produce credible FPS numbers already.

Not to say a standard isn't a worthy thing to have, but that it might get neglected. Just doing the testing on a selection of common mods & blasters would be a more well-read thing.
 

joneill809, on 19 Jun 2017 - 4:22 PM, said:

I haven't gotten into flywheels [much] yet. My kiddos are young and I enjoy modding blasters they can use so I focused on springers since I'm leery about having them playing with lipos right now.

Your kids can use 10kg modded retaliators...?

A stryfe or similar would be easier to use since it's just a trigger pull. That said, my kids don't touch my strayvan and largely use TTGs. Lipos are less scary than they seem, rig up an alarm and/or don't let them use the batteries when you aren't around. Or use a different kind of battery - I always forget what it is but NiMh or NICAD or something is supposed to work similar to Lipo with minor power reduction but is vastly safer.
 

joneill809, on 19 Jun 2017 - 4:22 PM, said:

I too am intrigued by the SCAR barrel. I have been messing with a design that would slip on to the Caliburn. I am starting without a twist. I've read the improvement in accuracy may be from a stabilizing effect rather than the twist so I added 4 parallel ridges to this design and some exhaust holes (it looked cool but maybe they are not needed - I can remove them in another design).

I'd asked Specter about the SCAR on Reddit the other day. He posted a video comparing straight to twisted on r/HPAnerf and said it was more accurate twisted. In that thread I got into an interesting discussion with Thunderkrunk (sp) and this is my understanding of the topic:

You want 6 flutes - they make a hexagon which better centers a sphere than a square 4-flute

You can achieve the same effect with porting, but TBH it sounds like way more work when the fishing line method works.

My understanding of the theory isn't that the spin helps the dart *outside* the barrel, but that it helps while it's still *inside* the barrel. It centers the dart within the blast force propelling it and vents excess around the dart evenly, causing the dart to maintain a desirable orientation, thus increasing accuracy, on leaving the barrel.

joneill809, on 19 Jun 2017 - 4:22 PM, said:

I also wanted everyone's feedback on this thread before I did a distance profiling on the Caliburn since that work is pretty time consuming (500 shots or more!).

Do it. I'm very interested in the accuracy/distance measurements you've done, even if your FPS numbers are low (and I'm not saying they are or are not, but to me your numbers sound more reliable because of your setup).

[joneill809]

Meaker VI, on 19 Jun 2017 - 6:38 PM, said:

Ok - I see how that'd eliminate machine/operator error in data collection. Again, I think you're probably doing it *better* than most, hence your results are more realistic (not all chrono posts go through 100 darts and take the average!).

 

Yeah the 100 shot count was more about trying to get a statistically significant population of shots at greater ranges. I still wanted the same number of shots at each range fired. I can probably reduce the count at close range, but I also like seeing the outliers that show up. I also wanted to do some random sampling from the 100 (e.g. random groups of 10) and show how the average can vary a good bit in such small populations. I thought that kind of plot would be helpful to show how expanding your sample count helps narrow your confidence intervals and give you a better representative average.

 

 

Meaker VI, on 19 Jun 2017 - 6:38 PM, said:

I'm concerned that the trick would be *actually getting people to really do the setup*. I don't think most nerfers would adhere to any standard created, and the false FPS data will continue to circulate. Encourage proper methods anyway, but post new data.

 

Agree. Even if someone can include 20-30 shots from a "standard blaster", that's good enough to get a baseline for a setup. Yeah I'll keep posting data as I get it! 

 

Meaker VI, on 19 Jun 2017 - 6:38 PM, said:

You wouldn't need to test *every* blaster with *every* configuration either - some blasters either aren't worth testing or are the same as other blasters, some setups aren't worth testing (unavailable) or are similar to other setups, and people doing unique stuff are either doing it for their own sake or to push the envelope. People pushing the envelope usually are the ones who have the means to produce credible FPS numbers already.

 

I see where you're going. Hopefully the Retaliator stuff is interesting - I find them a fun platform to work with. I have additional data on the Artifact Punisher kit too that I haven't plotted up all purty yet. I'll get to the stryfe's at some point - after I get more springer data collected. I really need to finish my EPT project too.

 

Meaker VI, on 19 Jun 2017 - 6:38 PM, said:

Your kids can use 10kg modded retaliators...?

A stryfe or similar would be easier to use since it's just a trigger pull. That said, my kids don't touch my strayvan and largely use TTGs. Lipos are less scary than they seem, rig up an alarm and/or don't let them use the batteries when you aren't around. Or use a different kind of battery - I always forget what it is but NiMh or NICAD or something is supposed to work similar to Lipo with minor power reduction but is vastly safer.

 

Haha no he sticks to his tri-strike for the neighborhood battles and I prime the modded Retaliators for him when he's plinking cups. I will likely roll one of them back to a 2.5 or 5 kg spring. But there's nothing like the grin of a seven year old wielding a Caliburn pounding a hard tip stefan into the fence at 70 feet. "Dad I can't blast people with this one, right?" He does like the stryfe, and he got the hang of rev, wait, pull pretty quickly. I've been reading that other thread about lipo alternatives and I may give that a go. I just don't feel comfortable letting him roam around unsupervised (the neighborhood kiddo wars often happen over the course of block or two) worry about voltages.

 

Meaker VI, on 19 Jun 2017 - 6:38 PM, said:

I'd asked Specter about the SCAR on Reddit the other day. He posted a video comparing straight to twisted on r/HPAnerf and said it was more accurate twisted. In that thread I got into an interesting discussion with Thunderkrunk (sp) and this is my understanding of the topic:

You want 6 flutes - they make a hexagon which better centers a sphere than a square 4-flute

You can achieve the same effect with porting, but TBH it sounds like way more work when the fishing line method works.

My understanding of the theory isn't that the spin helps the dart *outside* the barrel, but that it helps while it's still *inside* the barrel. It centers the dart within the blast force propelling it and vents excess around the dart evenly, causing the dart to maintain a desirable orientation, thus increasing accuracy, on leaving the barrel.

 

Thanks for that. I'll work up a few more models to test out based on this. I do have a line based barrel from Monkee Mods that I need to test out. I just haven't spent time figuring out how to mount it best. I'm interested in the 3D printed barrel attachments because they're a good simple model to start with as I get started with my printer. Seemed like an easier project.

 

Meaker VI, on 19 Jun 2017 - 6:38 PM, said:

Do it. I'm very interested in the accuracy/distance measurements you've done, even if your FPS numbers are low (and I'm not saying they are or are not, but to me your numbers sound more reliable because of your setup).

 

I will - I just wanted to get some feedback, and a SCAR / SCAR-like attachment or two printed up so I could gather some results concurrently and maybe work back the distances if the accuracy is good enough. I can also add in some standard length darts since the Caliburn can fire those - the Worker and Artifact kits I was messing with are stefan only so I started there just to compare. I like the mod "story" the plots show as you move through the different kits and springs, though I could use a break from unscrewing / rescrewing retaliators together. Now that I have the different springs characterized I can focus more on the distance studies which are a lot more work collecting data, but less modding / switching out components. 

[jwasko]

Coincidentally, I work in a Chem lab, too. You sound like one of those annoying QA types 

 

I was going to suggest that lighting may be causing issues with the chrono, but if you bought the lighting kit that shouldn't be it. You might still want to do some testing outside (on a cloudy day I think is best? It should say in the manual) and compare results.

 

A lot of environmental things can impact springer performance including temperature and humidity; they can cause the darts to shrink or swell. If the dart doesn't fit the barrel correctly, performance will be impacted. I suppose there could be issues with orings, too.

 

Dart weight and dart head shape can cause big differences too. I've seen a lot of people use elite darts with flywheels possibly because they give higher numbers, but they lack accuracy when it comes time to use them in battle.

 

A flywheel blaster, though it may be somewhat less consistent from shot to shot, might be a better baseline from nerfer to nerfer. Now, Hasbro may (or may not) have used slightly different motors from one production run to another, but it's at least a start. You may be familiar with Foam Data Services. Here is a video with some data on a stock stryfe:

 

Unfortunately I haven't found a central repository of all his data, but he links a spreadsheet in the description of each video he produces.

 

If you replace the motors and battery that will you another datapoint. There should be a good amount of data around for Rhino motors on 3S LiPo for instance (although I totally get your reluctance for LiPo).

 

 

This still leaves us with trying to correlate the data from your chronograph to the data from other people's chronographs. At my job, my analytical instrument is calibrated at least daily using NIST traceable standards purchased from two different sources. However, in order to make sure there isn't drift over time we analyze samples of known concentration. This concentration may be known to us (run as a Laboratory Control Spike or LCS) or tested blind (in the case of Proficientcy Tests that we must pass in order to stay certified with accrediting agencies).

 

Of course all Nerf blasters' FPS are only being tested by hobbyists using hobbyist equipment much like yourself (at best, but probably much worse!) so there is no real way to have a certified blaster with a definitely known FPS.

 

However, there is also the round robin test. If a single blaster (not multiples of the same blaster) is tested using several nerfers' chronos, then one might be able to determine whether there is a bias in your chrono and or other chronos. If there are enough participants, then you may be able to use some statistical wizardry  to come up with a number to use as a "true value" for the FPS for that blaster. Of course this would be easier if you met up with several people with chronos at a war or something. Otherwise you would have to mail the blaster from one person to the next.

 

Again, I sort of feel like a flywheel blaster may be less likely to get damaged en route/have its performance significantly changed over time. However, keep in mind that over time foam from the darts tends to build up on flywheels. Performance (FPS) can increase significantly during the initial buildup.

 

Apologies for the rambling post; been working on it in spurts for half the evening.

 

PS: Welcome to Nerfhaven, glad to have you!

[CaptainSlug]

jwasko, on 20 Jun 2017 - 02:01 AM, said:

I was going to suggest that lighting may be causing issues with the chrono

I can't stress this enough. Trying to use a chrono indoors under fluorescent lighting will result in overly-optimistic data. They require a steady non-flickering light source in order to read accurately.

[joneill809]

jwasko, on 20 Jun 2017 - 02:01 AM, said:

Coincidentally, I work in a Chem lab, too. You sound like one of those annoying QA types 

Well you got me there! I spent two years in a chem lab in grad school then I got into the software business and now I handle scalability and accuracy testing at-scale and workload modeling for backend software products nobody cares about ... until a cranky customer calls ...

 

jwasko, on 20 Jun 2017 - 02:01 AM, said:

I was going to suggest that lighting may be causing issues with the chrono, but if you bought the lighting kit that shouldn't be it. You might still want to do some testing outside (on a cloudy day I think is best? It should say in the manual) and compare results.

 

CaptainSlug, on 20 Jun 2017 - 02:06 AM, said:

I can't stress this enough. Trying to use a chrono indoors under fluorescent lighting will result in overly-optimistic data. They require a steady non-flickering light source in order to read accurately.

Great I should have posted a few months ago before the Texas summer hit; now I'll have to wake up at the crack of dawn to get outside as the temperature drops below 80! My test setup is in a bay window that gets mostly indirect natural light most of the day, and I'm using the lighting kit so I think I'm OK on light but it doesn't hurt to give it a go outdoors.

 

jwasko, on 20 Jun 2017 - 02:01 AM, said:

A lot of environmental things can impact springer performance including temperature and humidity; they can cause the darts to shrink or swell. If the dart doesn't fit the barrel correctly, performance will be impacted. I suppose there could be issues with orings, too.

 

Dart weight and dart head shape can cause big differences too. I've seen a lot of people use elite darts with flywheels possibly because they give higher numbers, but they lack accuracy when it comes time to use them in battle.

Yeah that's a lot of variables to consider. I think the reproducibility study I eluded to would tackle multiple variables concurrently. I could repeat the same test over the course of a few weeks and monitor the variability of the average / percentiles / std dev and get a better feel for the results I'm getting. Now that I've got the Caliburn going I'll focus on that for a while. It's fun to shoot so I don't mind piling up a thousand samples! I also grabbed a batch of elites out of my storage bin today. I'll start running those as well. Indoor. Outdoor. Time of day. Weather conditions. The problem with this approach is if we do uncover an anomaly then we have to dig into the individual variables more deeply, but we can cross that bridge...

 

jwasko, on 20 Jun 2017 - 02:01 AM, said:

A flywheel blaster, though it may be somewhat less consistent from shot to shot, might be a better baseline from nerfer to nerfer. Now, Hasbro may (or may not) have used slightly different motors from one production run to another, but it's at least a start. You may be familiar with Foam Data Services. Here is a video with some data on a stock stryfe:

 

Unfortunately I haven't found a central repository of all his data, but he links a spreadsheet in the description of each video he produces.

 

If you replace the motors and battery that will you another datapoint. There should be a good amount of data around for Rhino motors on 3S LiPo for instance (although I totally get your reluctance for LiPo).

Thanks! I'll start poking around his data sets too. I'll start getting my data in a format everyone can read and I'll dump it on Google Drive - I've got to noodle some on how best to organize it. Yeah I imagine Rhinos / 3S could be a plausible reference platform.

 

jwasko, on 20 Jun 2017 - 02:01 AM, said:

This still leaves us with trying to correlate the data from your chronograph to the data from other people's chronographs. At my job, my analytical instrument is calibrated at least daily using NIST traceable standards purchased from two different sources. However, in order to make sure there isn't drift over time we analyze samples of known concentration. This concentration may be known to us (run as a Laboratory Control Spike or LCS) or tested blind (in the case of Proficientcy Tests that we must pass in order to stay certified with accrediting agencies).

 

Of course all Nerf blasters' FPS are only being tested by hobbyists using hobbyist equipment much like yourself (at best, but probably much worse!) so there is no real way to have a certified blaster with a definitely known FPS.

 

However, there is also the round robin test. If a single blaster (not multiples of the same blaster) is tested using several nerfers' chronos, then one might be able to determine whether there is a bias in your chrono and or other chronos. If there are enough participants, then you may be able to use some statistical wizardry  to come up with a number to use as a "true value" for the FPS for that blaster. Of course this would be easier if you met up with several people with chronos at a war or something. Otherwise you would have to mail the blaster from one person to the next.

 

Again, I sort of feel like a flywheel blaster may be less likely to get damaged en route/have its performance significantly changed over time. However, keep in mind that over time foam from the darts tends to build up on flywheels. Performance (FPS) can increase significantly during the initial buildup.

The round robin test seems like an intriguing solution. I'm game for something like that. Of course I could also monitor for a price drop on another chronograph to compare those readings from these entry level devices. I have another message into the manufacturer asking about any recommendations they may have on test methodologies. We'll see if I hear back.

 

jwasko, on 20 Jun 2017 - 02:01 AM, said:

Apologies for the rambling post; been working on it in spurts for half the evening.

 

PS: Welcome to Nerfhaven, glad to have you!

Thanks! I appreciate the warm welcome! The discussion is also fun and a good diversion from my day job data sets!

[shmmee]

If you're doubting your chrony why not test it against the calibrated standard of gravity? All falling things accelerate at 32.2 feet per second (minus drag) so why not drop something dense and smooth surfaced (like a marble, ball bearing or heck, even a roundish rock might work) from a measured distance and compare the measured speed with the calculated speed?

 

We (the collective NIC) might even want to develop and accept our own standard to test and compare our chronys for accuracy. It'd be as simple as "drop a dense roundish object through it from a measured distance of 4'.  "? That's a test all chrony owners can perform without having to ship anything anywhere.

Edited by shmmee, 20 June 2017 - 08:56 AM.

[TriggerWarning]

I'd be careful about touting around the term accuracy though. Generally you want to measure spread. A few youtubers (bobo comes to mind) have measured by just recording the spread by camera. This chrono drop is no way to measure a blaster's accuracy. It's very one dimensional.

 

Additionally, I think your numbers for Claiburn are a little low. My snap bow can go up to 200s and caliburn as I understand it is supposed to be optomized to be much higher than 170s, since we're both using [k26]'s

 

And if you're printing a flute stabilizer, I'd probably want to print it sideways so the grain of the print doesnt grind against your outcoming darts. Do you have any photos of your stabilizer unit irl?

Edited by TriggerWarning, 20 June 2017 - 12:41 PM.

[joneill809]

shmmee, on 20 Jun 2017 - 1:49 PM, said:

If you're doubting your chrony why not test it against the calibrated standard of gravity? All falling things accelerate at 32.2 feet per second (minus drag) so why not drop something dense and smooth surfaced (like a marble, ball bearing or heck, even a roundish rock might work) from a measured distance and compare the measured speed with the calculated speed?

 

We (the collective NIC) might even want to develop and accept our own standard to test and compare our chronys for accuracy. It'd be as simple as "drop a dense roundish object through it from a measured distance of 4'.  "? That's a test all chrony owners can perform without having to ship anything anywhere.

I suppose I could give that a go just for kicks. I've been thinking it may be easier to just get a second Chronograph . 

 

TriggerWarning, on 20 Jun 2017 - 5:37 PM, said:

I'd be careful about touting around the term accuracy though. Generally you want to measure spread. A few youtubers (bobo comes to mind) have measured by just recording the spread by camera. This chrono drop is no way to measure a blaster's accuracy. It's very one dimensional.

 

Additionally, I think your numbers for Claiburn are a little low. My snap bow can go up to 200s and caliburn as I understand it is supposed to be optomized to be much higher than 170s, since we're both using [[[[[[[k26]]]]]]]'s

 

And if you're printing a flute stabilizer, I'd probably want to print it sideways so the grain of the print doesnt grind against your outcoming darts. Do you have any photos of your stabilizer unit irl?

The spread concept is interesting, albeit tougher for me to characterize. I was trying to think of a way to collect a larger population of data, and manually counting on screen would limit the number of data points and variety of test conditions I could capture, I think (at least it's a method I'd have to spend a good bit more time refining). I was thinking of some kind of target similar to an electronic dart board would be an interesting approach to registering spread. Right now though I was just focused on what I could do with the chronograph since I'm just getting started. I was interested in the fps numbers at a distance, and figured it would be interesting to couple that with a metric of how many shots it took to get to the population of results. I was thinking that IF a stabilizer helped, I should see a higher percentage of readings in my chrony, and I could measure the impact on fps at that distance. 

 

I'll get some photos of the stabilizers in the next few days. I printed two, one with 6 straight flutes and another with 8 straight flutes. I printed them vertically, so I can try another pass horizontally. I have not run them through the chronograph yet as I was looking to use the distance study as the main testing method and that will take a while.

 

In terms of fps, it does seem like most of my numbers are low across the board by 15% - 20%, so I am questioning the accuracy of the chronograph or my method. I was suspicious after the stock retaliator numbers, then I finally decided to post because of the Caliburn numbers. Captain Slug's numbers are in the 200's, and my readings are consistently lower. Speaking of, here's a few more data points from some other stefans I have:

 

Those worker darts look weird, but man were they consistent. It was followed by the Artifacts, which were all over the place. I did a second run with the hard tips since I had data with them already. I'll gather a few more reproducibility runs of the hard tips and the Workers to see how the numbers hold up. Overall the Caliburn is consistent for me, but low on my setup. I have to try different conditions (like outside) to see if these numbers shift.

 

Thanks for all the input! My test plan keeps growing...

[jwasko]

shmmee, on 20 Jun 2017 - 1:49 PM, said:

If you're doubting your chrony why not test it against the calibrated standard of gravity? All falling things accelerate at 32.2 feet per second (minus drag) so why not drop something dense and smooth surfaced (like a marble, ball bearing or heck, even a roundish rock might work) from a measured distance and compare the measured speed with the calculated speed?
 
We (the collective NIC) might even want to develop and accept our own standard to test and compare our chronys for accuracy. It'd be as simple as "drop a dense roundish object through it from a measured distance of 4'.  "? That's a test all chrony owners can perform without having to ship anything anywhere.

Gravity is such an awesome idea. Major props.

Something like slingshot ammo is something that should be fairly consistent and sufficiently dense that I don't think it would be significantly affected by drag or wind. The only issue might be release; perhaps a pin in a piece of PVC.

After calibrating with a slingshot ammo, marble, or whatever, you could also try dropping a dart (which would probably fall significantly slower).

 

Edit: I don't know if there is a lower end to what your chrono can read, but running some math dropping an object from 6ft above the chrono will only produce an FPS of 19.6 by the time it gets to the chrono. Getting closer to the FPS of your retaliator would require something like 25ft (40.1FPS)

 

Also, regardless of your chrono's capabilities, there will be additional acceleration between the firast sensor and the second sensor. If they are a foot apart (I have no idea what this length actually is), the velocity would change by ~4% between the sensors at around 20FPS. That change would decrease to ~1% at 40FPS.

Edited by jwasko, 22 June 2017 - 09:59 AM.

[joneill809]

I've been thinking about the gravity test and had the same concerns - achieving a high enough fps would require a significant drop distance. It can be done, but not easily, at least not easily at my house. 

 

I broke down and picked up a Competition Electronics unit. Figured it was worth the investment when I considered how much time I'd spend trying to validate my results using alternative methods. I fired a few shots through it with the Caliburn and got ~170 fps, then took both chronographs outside and got a few more readings in the same range. I need to get a better population of results but I need to think of what else I could be doing wrong. I am using NiMH rechargeable 9v's (I'm trying to be green!). I'm thinking of buying a few alkaline batteries to see if that changes anything. These 170 fps readings on two different units along with the other data sets that appear to come in 20% low seem to point to something systematic I'm doing (I think) so I have to think about what else I could be doing differently. The batteries were the only other thing I could think of right now.

 

Another interesting approach that I can try is a video of a few shots coupled with this: http://physlets.org/tracker/. At least it's a completely different method for validating fps and I think that will be next in the queue following the alkaline battery test. I did a cursory investigation on doppler based chronographs (man does the LabRadar look cool) but they start at higher fps ranges that would omit our entry level blasters / baseline. The LabRadar can track a projectile down range and provide fps readings along the way. It's not cheap but that is pretty awesome. 

 

On a separate note, here's a photo of the two vertical prints of the stabilizers I printed. I tried the 8 flute in a horizontal print without supports but it didn't print all the flutes correctly. I'm trying another horizontal pass with 6 flutes rotated with supports, however if I use supports internally I'll still have to sand/file the flutes. Any tips on printing orientation are appreciated. I'm totally spitballing here and just having fun, so please don't hesitate to correct any of my very basic design assumptions. These first attempts were me just goofing around with tinkercad just trying to figure out how to design models. 

 

 

The flutes stop 20mm from the back of the stabilizer (100mm). The flute tip ID is the same as the Caliburn barrel ID, and the ID of the rear of the stabilizer is the OD of the Caliburn barrel so I can get a friction fit. I don't know if the vent holes on design 05 (6 flute) are necessary - they just looked cool  I removed them in design 06 which had 8 flutes. Figured I could mess with the vent / no vent concept along with flute count. I started with straight flutes figuring I could add a turn later. I know others have tried the rotation / no rotation concept, but I was interested in applying whatever chronograph methodology I came up with using that testing methodology to get a broad range of data sets. Again, I'm just messing around to get familiar with design concepts, not trying to say I know what I'm doing...

[TriggerWarning]

Personally, what do you think of your fluted nonrotating barrel attachments as far as accuracy goes?  Is it something you'd realistically use in a war?

Edited by TriggerWarning, 07 July 2017 - 05:00 PM.

[Meaker VI]

TriggerWarning, on 07 Jul 2017 - 10:00 PM, said:

Personally, what do you think of your fluted nonrotating barrel attachments as far as accuracy goes?  Is it something you'd realistically use in a war?

Specter tested those on Reddit/r/hpanerf for me recently, apparently they don't have the same effect. I think Thunderkrunk and Toruk both have similar results to the SCAR setup by porting barrels though.

[joneill809]

TriggerWarning, on 07 Jul 2017 - 10:00 PM, said:

Personally, what do you think of your fluted nonrotating barrel attachments as far as accuracy goes?  Is it something you'd realistically use in a war?

I can't really answer your war question - I'm new to the hobby, and haven't been to one. My kids and I mess around with the blasters in the yard, and knock down cups, so I can't really tell you how war practical they are, but they seem to be popular in the SG nerfing community. I have a SCAR barrel from Monkee Mods and it's fascinating how they are tied, but it seemed like there could be a lot of variation in the build process. I liked the idea of trying to come up with a set of printable attachments (perhaps more consistent than strings?) that I could swap on and off for testing and it gave me a mechanism to tinker with our new printer. That's kind of where I am...still.

 

I don't have enough shots at a distance through the first barrels I printed to have formed an opinion. My initial tests focused on the friction fit of the barrel then I got side tracked with the boring baseline stuff. As of 2 days ago, I'm setup with two different models of chronographs, both with the ability to log data to my phone. This will support my requirement for larger shot populations (I'm not writing this stuff down - auto logging!). I want to do a baseline comparison on a single blaster (likely the Caliburn) across both chronos to validate my initial readings off (they correlated fairly well with limited shot counts). Once I'm confident that I'm getting consistent readings and my base blaster yields reproducible results, I'll move over to my distance tests and get those baselines collected.

 

After that I'll run the straight flute stabilizer tests from a distance and see how that goes. I still don't have a great mechanism to measure spread, but I think I'll tackle that in a follow on study. My first pass will use the same methodology I presented above which focused on whether or not I got a reading on the chronograph at various distances. This requires hitting a fairly small target consistently, and if my population is big enough, I should be able to measure the impact of various barrel designs. Anyhow that's the current plan.

 

 

Meaker VI, on 08 Jul 2017 - 01:57 AM, said:

Specter tested those on Reddit/r/hpanerf for me recently, apparently they don't have the same effect. I think Thunderkrunk and Toruk both have similar results to the SCAR setup by porting barrels though.

I just skimmed through the reddit post - I'll have to take a closer look at the references and see what data sets may be available for comparison. I haven't really done a broad search for existing materials so these references are really interesting and helpful. (Nice Caliburn print BTW!)

[Meaker VI]

joneill809, on 08 Jul 2017 - 03:55 AM, said:

perhaps more consistent than strings?

The strings should actually be pretty consistent provided they're tight enough - I'd think more consistent than prints if you specify which weight/brand of fishing line you're using and it's in a brass or aluminum barrel. Tightening should be easy to accomplish - I drew up a coupler I have yet to test that would slide over a barrel and capture the strings to pull them taught and allow you to twist them however far you wanted.
 

joneill809, on 08 Jul 2017 - 03:55 AM, said:

After that I'll run the straight flute stabilizer tests from a distance and see how that goes. I still don't have a great mechanism to measure spread, but I think I'll tackle that in a follow on study. My first pass will use the same methodology I presented above which focused on whether or not I got a reading on the chronograph at various distances. This requires hitting a fairly small target consistently, and if my population is big enough, I should be able to measure the impact of various barrel designs. Anyhow that's the current plan.

 I think your current setup, while labor intensive, is actually pretty good. The darts tend to curve, not just miss due to bad aim, making hitting a chrono-sized target is a hard task. It's at least objective and repeatable, so those are two big pluses.
 

joneill809, on 08 Jul 2017 - 03:55 AM, said:

I just skimmed through the reddit post - I'll have to take a closer look at the references and see what data sets may be available for comparison. I haven't really done a broad search for existing materials so these references are really interesting and helpful.

Toruk's stuff is usually pretty well researched and documented, and Thunderkrunk seems to have an interest into this stuff so he's provided excellent research material.
 

joneill809, on 08 Jul 2017 - 03:55 AM, said:

(Nice Caliburn print BTW!)

Thanks!

[joneill809]

It's been a while but I think I've made significant progress in establishing my confidence in my collection methodology. I made a few additions to my Nerf collection which has helped me validate my original numbers and hopefully establish a decent reference blaster for carrying forward my experiments, where I can inject regular testing of the reference blaster in between experimental runs. 

 

First up, I added a second chronograph setup. My original setup was a Caldwell Ballistic Precision Chronograph (CBPC) with the indoor light kit. This unit comes with a data cable that allows for data collection via a phone app. Given the uncertainty in the readings I was getting, I picked up a Competition Electronics ProChrono (CEP) and the C.E. Digital Link Bluetooth Adapter.

 

Initial testing indicated the CBPC and the CEP were reading within similar ranges. I didn't have time for a full session with both until tonight. What I found was the CBPC reported slightly higher FPS readings for the same blasters compared to the CEP. What blasters you ask? Well, with the prime sales I picked up 2 Apollos and 1 Artemis on sale. I thought (besides being cool) having a different blaster line with different projectiles would be an interesting reference blaster to add to the mix. The highlights of what I found were:

1. The Apollos were incredibly reproducible in their velocities. I chronographed both units on the CBPC, firing 100 shots each. Apollo 1 yielded 102 +/- 6 fps, while Apollo 2 yielded 102 +/ 4 fps. This was in range with velocities I have seen reported by others. I had not seen consistency like this from other blaster platforms in my work, so my current thinking is I will use these for a documenting the reproducibility of my setup as a long term reference blaster.
2. The Artemis came in lower at 92 +/- 12 fps on the CBPC (I'm providing 95th percentile confidence intervals for the populations here). This too was in range for what other folks have reported.
3. The CEP yielded 99 +/- 6 fps for the Apollo 1, and 87 +/- 12 for the Artemis. Looking at the confidence in the mean, the intervals are 99 +/- 0.5 (CEP) versus 102 +/- 0.5 (CBPC) for the Apollo 1, and 87 +/- 1.1 (CEP) versus 92 +/- 1.1 (CBPC). So the CEP is reporting 5 fps lower for the Artemis, and ~3 fps lower for the Apollo. This was interesting to note the variability in readings potentially introduced by the chronograph in use.
4. After 500 Rival rounds, I ran 100 Worker stefans through my Caliburn and got 170 +/- 19 fps for the population (170 +/- 1.9 for the confidence in the mean). I saw more variability this time after I had a jam and pinched a stefan in the barrel with the ram, so I'll have to take a closer look at what I may have done to drive up the confidence band around the population, but the average I saw were in line with my prior results.

So I'll give the Caliburn a once over and begin my distance testing, as I've satisfied myself that I have a reproducible configuration, and a reference blaster that's coming in with similar results that have been reported by the hobby. For those that like overly complicated plots, here are the details I logged in my spreadsheet where I'm tracking this stuff. I'd like to be able to share the full populations of results, so I'm tinkering with layouts. These are two detailed views (not sure how they'll render in the post so they will be redundant). Here's the top summary of how I'm logging data which only has the first 5 shots from each data set included (you can see the clearing of the jam remnants in the Caliburn's low fps shot):

 

And this is the full view with the entire population of shots (you should be able to click on it to see the full size image):

 

The gray bar charts are a new addition that help me quickly see the variability of a blaster's fps readings over the course of a chronograph session. You can see the "bumpiness" of the Artemis compared to how much more consistent and flat the Apollo plots are. 

 

My next step is to dive into the distance profiling I originally described. This will be a long go of it, starting with nailing down my methodology on the distance profiling. 

 

** EDIT **

 

Here's another set of plots from an X Zeus 2 session. I used the Ballistic Precision Chronograph for this one. I added this to show the bracketing of the session with the Apollo that I mentioned earlier. I start the session with 50 shots from the Apollo and end with it. This should provide the reviewer with figures for my reference blaster, and it should show any drift in my setup. In terms of the specific X Zeus data, man is this thing jamming on me. The only darts I could get through it consistently were the Artifact FVJ's, and I could only get them to load consistently by pointing the blaster down at the floor when priming. Anyhow, the main point of the edit was to show the bracketing of a reference blaster around a new set of tests.

 

 

(I can't seem to get the full image uploaded with the entire set of shots...I'll try again in another edit - it clearly shows the jams)

 

** EDIT **

 

OK I switched the format from PNG to JPG and it seemed to work. For those interested, you can expand the view of the file below and see the full shot sequences. Black shot FPS readings are errors reported by the chronograph. I remove the error codes so my percentile stats don't get messed up based on the formulas I wrote.

 

 

Edited by joneill809, 23 July 2017 - 10:36 PM.

[joneill809]

Let me know if this double post is out of line and I'll merge it, but I wanted to expand on the example of a reference blaster and I thought it warranted it's own post. I posted some data on the Caliburn thread, but I didn't want to dive into what I was doing with the reference blaster in conjunction with profiling the blaster under study on the bench. In this case I was collecting data for the Caliburn, but I bracketed the Caliburn with data from my reference blaster similar to what I did with the X Zeus 2 so that we had data from a known blaster available before and after my Caliburn collection. As I mentioned with the X Zeus, this gave me confidence in the setup, and the results that I was tracking for the Caliburn as the reference blaster results were consistent at the start and end of the run.

 

The added value of the reference blaster in this example was I also had the ability to compare historical Caliburn data to my latest data, along with historical data collected during the same bench session from my reference blaster. Here's that graphic from the Caliburn thread showing the use of the reference blaster bracketing:

 

I made one modification to this graphic, the addition of a slope value for the population of shots for each profiling run. The slope (your old friend rise over run) yields a rate of fps per shot (1 through n, which in this case was 100, or 50 in my one truncated run) that shows a trend in velocity over the collection period. This value shows you why I'm impressed with the Apollos, they are really consistent, and show no trending (close to zero slope) for the duration of their runs. My Caliburn collections are showing decreases in velocity over the course of testing. My current hypothesis is that my barrel seal is degrading over the course of collection, and I want to test that with an e-tape run to see if I can get a better seal. I'd like to get to see a more stable trend before moving forward with the more labor intensive distance profiling. 

 

I worked up this plot of the final Caliburn run to illustrate the slight downward trend I want to track down. Note the vertical axis is exaggerated (the decline is about 6 fps). I fitted a 10 point moving average as well as the linear regression summarized by the slope value in the graphic from above. I think this view illustrates the trend I "felt" I was seeing when I was collecting the data:

 

I'm not sure I would have noticed if I had stopped collecting after 20 to 30 shots. I definitely would not have noticed on a 10 shot sample. I think this is an interesting example of how trending over a fairly significant collection series can be observed more easily and possibly help debugging a setup. For comparison, here's the scatter plot for one of the Apollo runs (again with an exaggerated vertical axis):

 

It's really consistent and no observable long term trend. 

 

I've also been building up a series of profiling sessions from my reference blaster so that I can track the consistency of the bench setup over time. Here's an eye chart containing all my Apollo data (across two blasters and two chronographs):

 

This one just shows that over time I'll build up a significant set of data collections after which I can start experimenting with testing conditions and see what environment variables alter my results (e.g. moving outdoors under various lighting/temp conditions). I have one more e-tape test on my Caliburn to run through to try and chase down that downward trend in velocity, then I'm going to setup for a distance profiling run with it as I'm now satisfied that I'm getting results consistent with what Captain Slug sees.

 

To circle back to my original question, I'm settling on two approaches to comparing chronograph results from different hobbyists:

 

(1) Rival Apollos have shown promise as a reference blaster. Yeah, I've only tested two copies, but they are both really, really consistent. I'll wait for another sale and pickup another copy for testing, but I think they may be the "reference" we can use from site to site.

 

(2) We have to note what type / manufacturer of chronograph we're using. I've tested a Caldwell and Competition Electronics, and they consistently yield different results. I am biased toward the Caldwell (it's less expensive, and much faster at collecting data on my phone than the CE's bluetooth approach which misses about 10% of my shots fired if I don't pause for a second or two between shots), but the Caldwell reports slightly higher fps rates than the CE.

 

(3) I'm still in search of a high fps reference blaster. I don't think we can rely on corrections made based on results from reference blasters in the 50 to 100 fps range when profiling blasters in the 200+ fps range. These stock reference blasters like the Apollo are really useful, but I'd like to see what can be done about finding a stable reference at the 200 fps range. For example, the worker stefan darts were incredible with my first Caliburn run at 170 fps, but the tips started blowing off at 200 fps. Can I really apply correction factors from 70 fps ranges to 200+ ranges and assume those factors would hold? My gut tells me no, but I need more data to back that up. For now, I think we collect reference data, but I don't think you can apply corrections with confidence across that wide of a range of velocities. 

 

This may be old news to most of you folks, but I just find these kind of patterns fascinating and it's pretty cool to poke around these data sets 

[Meaker VI]

joneill809, on 29 Jul 2017 - 04:55 AM, said:

(3) I'm still in search of a high fps reference blaster. I don't think we can rely on corrections made based on results from reference blasters in the 50 to 100 fps range when profiling blasters in the 200+ fps range. These stock reference blasters like the Apollo are really useful, but I'd like to see what can be done about finding a stable reference at the 200 fps range. For example, the worker stefan darts were incredible with my first Caliburn run at 170 fps, but the tips started blowing off at 200 fps. Can I really apply correction factors from 70 fps ranges to 200+ ranges and assume those factors would hold? My gut tells me no, but I need more data to back that up. For now, I think we collect reference data, but I don't think you can apply corrections with confidence across that wide of a range of velocities. 
 
This may be old news to most of you folks, but I just find these kind of patterns fascinating and it's pretty cool to poke around these data sets 

The Caliburn is probably the only blaster consistent across fields that is also 200+FPS. There aren't many that aren't much more homebrew or are too niche to have coverage. SNAPS, Rainbows, ESLT's, +Bow's, etc. - these are all NH blasters, and very few have penetrated like Caliburn has into the superstock world. HPA/Pneumatics are, likewise, extremely variable - a slightly longer barrel and vent PSI and suddenly you've got a totally different blaster.

 

I really like the data you're putting together here, don't worry about posting as often as is necessary to update.

[joneill809]

Meaker VI, on 29 Jul 2017 - 4:31 PM, said:

I really like the data you're putting together here, don't worry about posting as often as is necessary to update.

Thanks! It's been an interesting project. I'm still trying to figure out the best way to present the data sets and some of the stats that I *think* are important. The plots are evolving a bit - I worked over the layout this weekend to condense some of the summary data and I'm sure it will continue to change. 

 

Meaker VI, on 29 Jul 2017 - 4:31 PM, said:

The Caliburn is probably the only blaster consistent across fields that is also 200+FPS. There aren't many that aren't much more homebrew or are too niche to have coverage. SNAPS, Rainbows, ESLT's, +Bow's, etc. - these are all NH blasters, and very few have penetrated like Caliburn has into the superstock world. HPA/Pneumatics are, likewise, extremely variable - a slightly longer barrel and vent PSI and suddenly you've got a totally different blaster.

Yeah, I'm spending a lot of time with the Caliburn because I think it will be a solid reference blaster once I get used to operating it consistently. I think a lot of the variability that I am seeing is just my learning curve (like lithium grease vs. silicone grease).

 

I ran two more sets of experiments this weekend. I did one wrap of e-tape around the Ram to see if the decline in fps I was seen was tied to the o-ring seal. Turns out it wasn't, and now I'm seeing slight drops in my reference blasters. I'm wondering if it's something I'm doing. Either way, it's about 1% on both the Apollo and Caliburn, so I'm not sure what to make of it yet. Here's the o-ring data bracketed by the Apollo reference blaster. In this case, I recorded 194 +/- 10, compared to 197 +/- 14, so no change. The decline in fps was -1.8% with the e-tape versus -3.2% without. Not sure of the significance of that yet, so I'll keep watching. Here's the data:

 

Next up, full length darts. I picked up some ACC-like darts ("Phat Welt Wackers") off Amazon. I also picked up some Koosh darts. I ran the Apollo, then runs of 100 PWW's, 100 Kooshes, 100 PWW's, then the Apollo. I collected the PWW's twice because of how lousy the Koosh performed and I wanted to rule out any platform issues. Here's the data set:

 

The error range on the second run of PWW's was impacted by a bad dart - if I drop the 123 round (tip was blown off the dart) then the fps reading was 200 +/- 12, nearly identical to the first run. So the koosh darts performed poorly, and my elbow needs a break after 500 rounds today

 

If you're interested in the plot designs, I tweaked the data layout a bit and tried to highlight the confidence interval more clearly. The + and - design is something I lifted from Tufte. I'm still trying to figure out if I like it better than a straight "201 +/- 12" but it kind of looks cool. The population statistics are presented below the average, and I moved the box plot like graphic up next to the stats that it represents. It's a bit redundant, but I like having the numbers to review, but I think the plots make for an informative rapid assessment across data sets. 

 

I'm trying two plots on top of each other. The solid gray band is a bar plot representing the shot velocities for each experiment. The vertical axis is consistent across all plots. The scatter plot is a representation of the same data in the bar plot, but with an exaggerated vertical axis tied to the min/max of the specific experiment. There are two curve fits to the scatter plot, a linear fit and a 10 point moving average. They are not terribly intuitive at this point, but once you know what to look for, they are a good tool for evaluating the data set quickly. I'll keep iterating on the design, so feedback is welcome!