2

I am running celery worker(version 4.4) on windows machine, when I run the worker with -P eventlet option it throws Attribute error. Error logs are as follows:-

pipenv run celery worker -A src.celery_app -l info -P eventlet --without-mingle --without-heartbeat --without-gossip -Q queue1 -n worker1
Traceback (most recent call last):
  File "d:\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\G-US01.Test\.virtualenvs\celery-z5n-38Vt\Scripts\celery.exe\__main__.py", line 7, in <module>
  File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\celery\__main__.py", line 14, in main
    maybe_patch_concurrency()
  File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\celery\__init__.py", line 152, in maybe_patch_concurrency
    patcher()
  File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\celery\__init__.py", line 109, in _patch_eventlet
    eventlet.monkey_patch()
  File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\eventlet\patcher.py", line 334, in monkey_patch
    fix_threading_active()
  File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\eventlet\patcher.py", line 331, in fix_threading_active
    _os.register_at_fork(
AttributeError: module 'os' has no attribute 'register_at_fork'

I have installed eventlet in virtual environment, the pipfile contents are as follows:-

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
rope = "*"
autopep8 = "*"

[packages]
eventlet = "*"
psutil = "*"
celery = "*"
pythonnet = "*"
redis = "*"
gevent = "*"

[requires]
python_version = "3.7"

Please let me know where I am going wrong.

sattva_venu
  • 677
  • 8
  • 22

4 Answers4

1

os.register_at_fork is a new function available since Python 3.7, it is only available for Unix systems (Source from Python doc) and Eventlet use it to patch threading library.

There is an issue opened in Eventlet Github: https://github.com/eventlet/eventlet/issues/644

So I don't think Celery -A app worker -l info -P eventlet is any longer a good alternative for Windows.

Personally I ran:

Celery -A app worker -l info

And it worked with Windows 10, Python 3.7 and Celery 4.4.7.

Antiez
  • 679
  • 7
  • 11
  • celery version 4 will not work with ```celery -A app worker -l info```, it gives error if we don't use -P option. Error log is ```ValueError: need more than 0 values to unpack``` – sattva_venu Sep 11 '20 at 08:50
  • 1
    Indeed. I succeeded to make it work with `celery -A app worker -l info -P solo` but from my understanding it removes concurrency properties of celery which is not really satisfying... – Antiez Sep 11 '20 at 09:55
  • 1
    Your command will work on windows if environment variable is set ```os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')```. With this no need to mention -P flag in celery. – sattva_venu Sep 20 '20 at 05:11
0

One reason you are facing this is because of the fact that Celery works on a pre-fork model. So if the underlying OS does not support it, you will have a tough time running celery. As per my knowledge, this model does not exist for the Windows kernel.

You can still use Cygwin if you want to make it work on windows or have Linux subsystem available for Windows

Sources:-

  1. How to run celery on windows?
  2. https://docs.celeryproject.org/en/stable/faq.html#does-celery-support-windows
onlinejudge95
  • 333
  • 3
  • 9
0

As hinted at by @金奕峰, os.register_at_fork was introduced in eventlet v0.27.0 c.f. commit compare on github. Specifying version 0.26.0 in your virtual environment's package list might solve the problem (for now).

André_K
  • 1
-1

is the eventlet version 0.26; pip install eventlet==0.26

金奕峰
  • 1