#966 new defect

document munin plugins and make them discoverable

Reported by: zooko Owned by: freestorm
Priority: major Milestone: soon
Component: documentation Version: 1.6.0
Keywords: easy docs statistics Cc:
Launchpad Bug:

Description

allmydata.com maintains some munin plugins to help you monitor the health and performance of your Tahoe-LAFS nodes. You can find them in misc/munin. But how are users supposed to discover this fact, and how do they learn how to use these munin plugins?

Attachments (6)

munin_stats-gatherer.darcs.txt (50.8 KB) - added by freestorm at 2010-05-30T20:58:02Z.
move_munin_plugins_files.dpatch (5.7 KB) - added by freestorm at 2010-08-01T20:22:49Z.
Split munin plugins into 2 directories
munin_plugins_for_stats-gatherer.dpatch (112.7 KB) - added by freestorm at 2010-08-11T20:14:23Z.
Patch for Munin Stats-Gatherer plugins
deep_manifest.php (2.0 KB) - added by secorp at 2010-09-12T15:30:03Z.
deep_size.php (1.6 KB) - added by secorp at 2010-09-12T15:30:16Z.
local_munin_plugins.darcs.patch (106.4 KB) - added by freestorm at 2010-10-18T20:02:38Z.
Munin local plugins

Download all attachments as: .zip

Change History (30)

Changed at 2010-05-30T20:58:02Z by freestorm

comment:1 Changed at 2010-08-01T14:11:35Z by zooko

Please use a more informative patch description, like the example patches shown here:

http://tahoe-lafs.org/trac/tahoe-lafs/wiki/Patches

comment:2 Changed at 2010-08-01T20:18:40Z by freestorm

  • Owner changed from somebody to freestorm

Changed at 2010-08-01T20:22:49Z by freestorm

Split munin plugins into 2 directories

comment:3 Changed at 2010-08-01T20:24:06Z by freestorm

Please ignore munin_stats-gatherer.darcs.txt patch

comment:4 Changed at 2010-08-05T22:45:41Z by freestorm

Is tahoe_estimate_files relevant?

I can't get it working with my config

I set:

node_dirs = [os.path.expanduser("/home/tahoe-storage/")]
N = 10
num_servers = 5

correction is always 0.0, so

files = unique_strings * (32*32/len(sections)) * correction

will always return files.value 0

comment:5 follow-up: Changed at 2010-08-06T21:49:47Z by freestorm

I need help from somebody to review the future README file for stats-gatherer plugins.

Please read and edit this wiki page
When file is going to be ready, I will create the patch.

comment:6 Changed at 2010-08-07T00:56:43Z by warner

tahoe_estimate_files assumes an allmydata-scale grid, in which the number of servers is somewhat larger than the number of shares (N), such that each server will have either 1 or 0 shares. With your values of N=10 and num_servers=5, each server will have 2 shares, and we'd need a different equation. I think for your environment you'd want to set correction to 1.0 to get it right (if every file appears on every server, there's no undercounting to be corrected for, so the chance/no_chance/correction stuff is irrelevant).

comment:7 Changed at 2010-08-07T09:42:25Z by freestorm

Thanks for your reply Warner,

I will modify tahoe_estimate_files to add plugin settings in tahoe-conf like other plugins

Changed at 2010-08-11T20:14:23Z by freestorm

Patch for Munin Stats-Gatherer plugins

comment:8 Changed at 2010-08-11T20:19:19Z by freestorm

Patch munin_plugins_for_stats-gatherer.dpatch is ready for review.
It contain modifications for Munin Stats-Gatherer plugins only
Another patch will contain "local_plugins" modifications

comment:9 Changed at 2010-09-12T13:14:49Z by freestorm

Munin plugin tahoe_overhead speak about a PHP script to collect total-deep-size from another server.

Somebody know where this PHP script can be found?

Changed at 2010-09-12T15:30:03Z by secorp

Changed at 2010-09-12T15:30:16Z by secorp

comment:10 follow-up: Changed at 2010-09-12T15:31:37Z by secorp

The last two attachments are the working copies of the php scripts. The checked in versions are in the www project on the old dev Allmydata box. We should really get this out into the public too ... I thought we had done this already but can't seem to find the trac instance we set up for it.

comment:11 in reply to: ↑ 10 Changed at 2010-09-13T19:22:33Z by freestorm

Replying to secorp:

The last two attachments are the working copies of the php scripts. The checked in versions are in the www project on the old dev Allmydata box. We should really get this out into the public too ... I thought we had done this already but can't seem to find the trac instance we set up for it.

Thanks for the files secorp

I looked them, but to fully understand them, I need the DB structure, .inc/includes.php and .inc/ui/dbconnection.php.

Actually I will mark tahoe_overhead as boken, because we need to close #669 before continue with those PHP scripts.

Last edited at 2010-09-13T22:43:01Z by freestorm (previous) (diff)

comment:12 Changed at 2010-09-13T21:07:05Z by warner

those PHP scripts are deeply obsolete, because they depended upon the DB that managed customer accounts at allmydata.com . I'd suggest adding a comment that says "to use this script, you will need some source of 'tahoe deep-size' results for all rootcaps that reference data in your grid. This tool was written for the allmydata.com environment, which maintained such a value."

comment:13 Changed at 2010-10-02T14:47:15Z by freestorm

I wrote the README file for munin local plugins.

It would be nice if someone can take a look a this wiki page to correct my spelling errors.

As far as this page is good I will write the Patch.

comment:14 follow-up: Changed at 2010-10-02T20:55:49Z by terrell

i've looked over the linked page in comment:13 above and made some whitespace and formatting/spelling changes.

i'm not sure of the content itself - there are underscores at the end of two of the plugin names...

the two examples at the end use a plugin that isn't listed in the table.

comment:15 in reply to: ↑ 5 Changed at 2010-10-02T21:13:41Z by terrell

Replying to freestorm:

I need help from somebody to review the future README file for stats-gatherer plugins.

Please read and edit this wiki page
When file is going to be ready, I will create the patch.

i've retouched this page as well.

comment:16 in reply to: ↑ 14 Changed at 2010-10-02T21:31:37Z by freestorm

Replying to terrell:

i've looked over the linked page in comment:13 above and made some whitespace and formatting/spelling changes.

i'm not sure of the content itself - there are underscores at the end of two of the plugin names...

the two examples at the end use a plugin that isn't listed in the table.

Thanks for your help. I'm going to edit the two example at the end of file.

The underscore at the end of some plugin is normal.

But I'm not sure that we need to put the README in wiki format. The final README format will be plain text. I use the wiki to review those files before creating the patch

Changed at 2010-10-18T20:02:38Z by freestorm

Munin local plugins

comment:17 follow-up: Changed at 2010-10-18T20:05:12Z by freestorm

Patch munin_plugins_for_stats-gatherer.dpatch is ready for review.
It contain modifications for Munin Stats-Gatherer plugins only

Patch local_munin_plugins.darcs.patch is ready for review.
It contain modifications for Munin Local plugins.

comment:18 Changed at 2010-10-18T20:06:04Z by freestorm

  • Keywords review-needed added
  • Owner changed from freestorm to someone

comment:19 Changed at 2010-10-19T02:59:56Z by zooko

  • Milestone changed from undecided to 1.8.1

We could consider including this in 1.8.1 because it doesn't touch core code and therefore can't introduce a regression.

comment:20 in reply to: ↑ 17 ; follow-up: Changed at 2010-10-30T11:06:15Z by francois

  • Owner changed from someone to freestorm

Hi Fred, I have reviewed your two patches, see my comments in-line.

Replying to freestorm:

Patch munin_plugins_for_stats-gatherer.dpatch is ready for review.

This statement is wrong, a stats-gatherer is required, not a helper. I guess you can remove this line because this is already stated previously in the document.

  • You should probably warn users in the NEWS file that tahoe_* plugins have been renamed tahoe_stats_*.
  • It seems that there is weird indentation between lines 773 and 776.

Patch local_munin_plugins.darcs.patch is ready for review.
It contain modifications for Munin Local plugins.

  • A small typo on line 27, Taoe-LAFS instead of Tahoe-LAFS
  • The directory mentionned in line 38 should be probably be specified relative to the top directory, ie. contrib/stats-gatherer_plugins/ instead of stats-gatherer_plugins. Same comment for all directories.

That's all, thanks for updating this documentation, I'm sure that new users of the munin plugins are going to have much less trouble getting started ;)

comment:21 Changed at 2010-10-30T11:06:29Z by francois

  • Keywords review-needed removed

comment:22 in reply to: ↑ 20 ; follow-up: Changed at 2010-10-31T10:13:51Z by freestorm

Replying to francois:

Hi François,

I have some questions about your comments.

Hi Fred, I have reviewed your two patches, see my comments in-line.

Replying to freestorm:

Patch munin_plugins_for_stats-gatherer.dpatch is ready for review.

This statement is wrong, a stats-gatherer is required, not a helper. I guess you can remove this line because this is already stated previously in the document.

I wrote this because if you don't have a Tahoe-LAFS Helper, it make no sense to run these plugins.
The helpers plugins would return 0 or NULL if it not exist a Helper on the grid (I need to make further tests to be sure)

  • You should probably warn users in the NEWS file that tahoe_* plugins have been renamed tahoe_stats_*.
  • It seems that there is weird indentation between lines 773 and 776.

Patch local_munin_plugins.darcs.patch is ready for review.
It contain modifications for Munin Local plugins.

  • A small typo on line 27, Taoe-LAFS instead of Tahoe-LAFS
  • The directory mentionned in line 38 should be probably be specified relative to the top directory, ie. contrib/stats-gatherer_plugins/ instead of stats-gatherer_plugins. Same comment for all directories.

Need I refer all directories from $TAHOE_SOURCE?
In the current Patch, all directories are mentionned from these files:

  • README
  • stats-gatherer_plugins/README
  • local_plugins/README


That's all, thanks for updating this documentation, I'm sure that new users of the munin plugins are going to have much less trouble getting started ;)

comment:23 in reply to: ↑ 22 Changed at 2010-11-01T09:14:18Z by francois

Replying to freestorm:

I wrote this because if you don't have a Tahoe-LAFS Helper, it make no sense to run these plugins.
The helpers plugins would return 0 or NULL if it not exist a Helper on the grid (I need to make further tests to be sure)

Oops, my bad, my comment was indeed incorrect, please discard it.

Need I refer all directories from $TAHOE_SOURCE?
In the current Patch, all directories are mentionned from these files:

  • README
  • stats-gatherer_plugins/README
  • local_plugins/README


Yes, it looks much clearer to me to always use the top directory as reference.

comment:24 Changed at 2010-11-02T02:32:25Z by zooko

  • Milestone changed from 1.8.1 to soon
Note: See TracTickets for help on using tickets.