7

I have a Plone site (something around 4.2.4, telling from a version.txt in the root directory) which I'd like to update to a recent version (I found this how-to) (as of now, 4.3.4); I have inherited a plethora of pinned versions which are not documented and might be outdated.

When commenting out my versions.cfg, using http://dist.plone.org/release/4.3-latest/versions.cfg instead and re-running buildout with -vvv, I get:

Develop: '.../src/collective.mathjax'
in: '.../src/collective.mathjax'
/tmp/tmpbXsnpD -q develop -mxN -d .../develop-eggs/tmp2yhe9ubuild
...
Installing 'zc.recipe.egg'.
We have the best distribution that satisfies 'zc.recipe.egg<2dev'.
Picked: zc.recipe.egg = 1.3.2
...
There is a version conflict.
We already have: zc.recipe.egg 1.3.2.
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe zc.recipe.testrunner.

So there seems to be some requirement for a sub-2 version of zc.recipe.egg, but I can't find it. (In my versions.cfg it is pinned to 2.0.1 - which used to work, surprisingly.)

I searched the tree with find and grep, looking for rc.recipe.egg in version*.cfg and setup.py files, but I couldn't find anything but in my project root directory. I even searched every single file below .../src/collective.mathjax, without success.

How can I ferret out this dependency? Thank you!

Tobias
  • 2,481
  • 3
  • 26
  • 38
  • Can you check what version of zc.recipe.testrunner you have? In my plone 4.3.4 buildout I have zc.recipe.testrunner pinned to 1.2.1, and zc.recipe.egg to 2.0.1 – sunew Apr 27 '15 at 12:04
  • @sunew: `zc.recipe.testrunner` is pinned to 2.0.0; both in my eggs directory and download-cache, I have 1.2.1 and 1.4.0 as well. – Tobias Apr 27 '15 at 14:01
  • you need to pin zc.recipe.testrunner to 1.2.1 in your versions.cfg – sunew Apr 27 '15 at 14:39
  • 1
    `versions.cfg` is pulling more version-configs via the `extends`-option, make sure to check them, too. – Ida Aug 19 '15 at 08:28
  • @Ida: Right, I found both `zc.recipe.egg` and `zc.recipe.egg` in http://dist.plone.org/versions/zopetoolkit-1-0-8-ztk-versions.cfg. (We don't have a tool which simply resolves the `extends` specifications and emits the flat configuration for grepping, right?) If you make this an answer, I'll accept it. – Tobias Aug 19 '15 at 08:47
  • @Tobias: Yes, we have now :-) (if I got your quest right) even several possibilities, see answer below. – Ida Aug 19 '15 at 09:54
  • @Tobias: Thanks for your offer, anything improvable in my answer of your side? – Ida Aug 24 '15 at 09:20
  • @Ida: I'd emphasize the recursion part a little bit more visibly; it should jump right into reader's face ;-) As for the tool - I'd like to have a tool which simply does a recursive dump of the buildout script (not limited to versions), resolving urls as needed, and possibly caching remote resources; perhaps I'll write it myself. I'd like to have something like that for Apache configuration as well, since the syntax check misses some fatal errors ... – Tobias Aug 24 '15 at 12:00
  • Huh, don't get immediately how an Apache's config would be involved here. Did you give M.v.Rees' script or mine a try? If you still feel to improve the answer and can see an edit-button on it, go ahead! (I should figure out, how the community-pages work, one day). – Ida Aug 24 '15 at 16:26
  • My approach is to have a dump script which simply resolves includes, like a `C` preprocessor does. I need it seldom, but then I miss it all the more. For the `buildout` module, I'll happily snitch existing code, with your kind permission ;-) – Tobias Aug 25 '15 at 07:17
  • @Tobias: Yeah I guess one could summarize it with "How to get an overview of all dependency-declarations?" (and detect possible circular-loops). There's another script I started once, to have it as an html-doc, but didn't finish: Looks in an existing eggs-cache for all newest eggs and lists their deps: https://github.com/ida/skriptz/blob/master/py/plone/getDeps/markup-generation/serial_egg_deps_lists.py You are permitted to use whatever you consider helpful, free as in DOM :-) (I would love to make this a wiki-thing, but erm, time is not on my side). – Ida Aug 28 '15 at 15:44
  • Just stumbled over https://pypi.python.org/pypi/z3c.dependencychecker, which also checks, if you have unneeded/outdated dep-declas :-) Got to look into that closer, too, one day. – Ida Sep 10 '15 at 12:58

5 Answers5

3

but a better way to upgrade the existing installation is probably to get a standard buildout for the plone version you want to upgrade to, and then add your non-standard eggs to this buildout. Finally move your database and blobs over to the new installation, and follow the upgrade guide.

sunew
  • 477
  • 5
  • 11
3

Basically there's three places to look for version-pinnings:

1.) The requires-files of eggs released on PyPi, like Luca Fabbri pointed out, which you can search for pins like this:

grep -r --include=requires.txt "dependency.to.search.for" path/to/eggs-cache

2.) The setup.py-files of development-eggs, similar searchable like:

grep -r --include=setup.py "dependency.to.search.for" path/to/dev-eggs-cache

3.) The [versions]-part of config-files, where in this case the version.cfg is pulling more version-configs in, via its extends-option and the pulled one's might also specify more configs via extends.

You're lucky, admired M. v. Rees has shared a snippet, on how to get all pinnings of all Plone versions: https://gist.github.com/mauritsvanrees/99cb4a25b622479e7dc3

Ida
  • 3,994
  • 21
  • 40
  • Yes, the recursion via `extends` options is the point; thank you! – Tobias Aug 24 '15 at 11:54
  • Circular loop it is, not recursion :-) – Ida Aug 24 '15 at 16:23
  • OK, I tried Maurits' script; it downloads the version pinnings of all Plone version up to a hard-coded newest version to the current working directory. This is a useful thing to have but doesn't help for requirements specified by the packages themselves. – Tobias Nov 14 '16 at 11:34
  • Oh, and of course we are talking about *recursion*. Circular loops can occur during recursion and must be handled, otherwise faulty input could make the process fail. – Tobias Nov 14 '16 at 11:37
  • Some additions since I just had such problems again: The `--include` argument to the `grep -r` command can be specified more than once (my version; at least it's worth a try), so the searches for `requires.txt` and `setup.py` can be done in one command. Furthermore, the wrong version ("We already have ...") can be in the `develop` area (`src/` directory) ... for `Products.GenericSetup`, the version spec was not in `setup.py` but in some `version.txt`. – Tobias Sep 29 '17 at 14:30
  • Can you see an edit-button? Feel free to complement and have a nice weekend! – Ida Sep 30 '17 at 10:07
2

The dependency is probably inside a 3rd party egg (so: no setup.py in it). Search again inside ./eggs/*/EGG-INFO/requires.txt (if your egg directory is inside the buildout root).

keul
  • 7,673
  • 20
  • 45
  • Sadly no success yet; only requirements without version, or with `>=1.2.0` or `>=1.3.0` found this way ... – Tobias Apr 23 '15 at 15:15
  • if you have defined a custom `eggs-directory` in your `~/.buildout/default.cfg` you'll need to search these eggs too. in addition the dependency might be in a zipped egg-file – fRiSi Apr 24 '15 at 08:34
  • @fRisi: There indeed is one zipped egg, but no luck - it doesn't contain the dependency ... – Tobias Apr 27 '15 at 08:07
  • 1
    Note for novices, the full command is: `grep -r --include=requires.txt "dependency.to.search.for" path/to/eggs-cache` IIRC, found this initially in one of Mikko Ohtamaa's exquisite posts: http://opensourcehacker.com – Ida Aug 19 '15 at 08:20
1

It could be that you have on your buildout configuration to not look for a newer version if you already have one locally. There should be a line like this:

newest = false

You could try to either remove your local cache of eggs or explicitly set to not use the global one and use a specific one (empty).

Something like:

[buildout]
eggs-directory = /home/USER/SOMEWHERE/eggs
download-cache = /home/USER/SOMEWHERE/downloads
extends-cache = /home/USER/SOMEWHERE/extends
gforcada
  • 2,498
  • 17
  • 26
1

You can use "eggdeps" (search pypi for it) to get a tree of all dependencies in your buildout - perhaps this can be helpful. Add the egg to your buildout and rerun buildout. Do it on your original, working buildout configuration, before making the changes you mentioned. (Generating the eggdeps script requires buildout to finish successfully).

add this to your buildout configuration:

parts +=
    eggdeps

...

[eggdeps]
recipe = zc.recipe.egg
eggs = tl.eggdeps
       ${instance:eggs}
scripts = eggdeps

Run buildout again. Now you have a script bin/eggdeps, that prints a tree of all dependencies. Run it:

./bin/eggdeps -n

Example output:

zope.app.pagetemplate 3.11.2
    setuptools 8.0.2
    zope.browserpage 3.12.2 ...
    zope.component 3.9.5 [hook] ...
    zope.configuration 3.7.4 ...
    zope.dublincore 3.7.0
        pytz 2013b0
        setuptools 8.0.2
        zope.component 3.9.5 ...
        zope.datetime 3.4.1 ...
        zope.interface 3.6.7 ...
        zope.lifecycleevent 3.6.2 ...
        zope.location 3.9.1 ...
        zope.schema 4.2.2 ...
        zope.security 3.7.4 ...
      [test]
        zope.annotation 3.5.0 ...
        zope.testing 3.9.7 ...
    zope.i18nmessageid 3.5.3 ...
    zope.interface 3.6.7 ...
    zope.pagetemplate 3.6.3 ...
    zope.schema 4.2.2 ...
sunew
  • 477
  • 5
  • 11
  • I installed eggdeps and ran it. It seems to work, but so far I was not able to produce any output yet which contains `zc.recipe` at all ... – Tobias Apr 27 '15 at 08:31
  • ah, sorry. I missed that your problem has to do with a recipe. Recipes etc. are only pulled in during buildout, and are not part of the final environment. Therefore they do not show up in eggdeps since that shows the eggs in the resulting pythonpath of the build. - Check my comment to your question. – sunew Apr 27 '15 at 12:00
  • The `eggdeps` hint is very useful in other cases anyway; I had such a use case just now. – Tobias Nov 14 '16 at 11:44