scripts: make output from run-all-cleanup easier to parse

Make it clear that it didn't end prematurely due to some invisible problem.

vcs: fix validation of Git https URLs (Issue #365)

Fix py3 migration bug.

hg: read everything from hgrc, without config section whitelisting (Issue #246)

The whitelisting seems pointless, is hard to maintain, and can't be customized.

Also, mercurial.localrepo.instance will read the full config file anyway.

hg: always show and run Mercurial hooks in alphabetical order (Issue #246)

Mercurial will generally run hooks in the order they are found in the
configuration. For entries found in the database, there is no such order.
Instead, always use alphabetical order for these.

Since we now want to order things explicitly in the db query, we want an index
with a composite key. We do that even though we don't really need it for the
few entries in this table, and even though it might/could use the same index as
the existing unique constraint. This composite UniqueConstraint was added in
b9f4b444a172 where it replaced a wrong UniqueConstraint that could/should have
been removed in c25191aadf92. Fix that while touching this area and running a
migration script.

hg: set HGPLAIN globally to minimize potential impact from reading hgrc configuration when creating ui

See `hg help scripting` for HGPLAIN description ... but Kallithea will invoke
Mercurial internals directly, so HGPLAIN shouldn't really impact anything.

scripts: drop isort --wrap-length 160 - it is broken with py3 and not really necessary

Under Python 3, isort 4.3.21 failed with
https://github.com/timothycrosley/isort/issues/741 on
kallithea/controllers/api/api.py :

Traceback (most recent call last):
File "data/env/bin/isort", line 10, in <module>
sys.exit(main())
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 379, in main
for sort_attempt in attempt_iterator:
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 377, in <genexpr>
attempt_iterator = (sort_imports(file_name, **arguments) for file_name in file_names)
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 88, in sort_imports
result = SortImports(file_name, **arguments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 207, in __init__
self._add_formatted_imports()
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 606, in _add_formatted_imports
self._add_from_imports(from_modules, section, section_output, sort_ignore_case)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 526, in _add_from_imports
import_statement = self._multi_line_reformat(import_start, from_import_section, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 552, in _multi_line_reformat
dynamic_indent, indent, line_length, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 705, in _output_grid
if len(next_statement.split(self.line_separator)[-1]) + 1 > line_length:
TypeError: '>' not supported between instances of 'int' and 'str'

validators: don't catch all Exceptions as invalid clone URIs, be specific

When adding a new repository with a remote clone URI, the URI will be
validated in some way. Several exceptions could occur during that
validation.

Previously, the code would catch based on 'Exception', which means that
_any_ exception would cause the URI to be found invalid. This means that
errors in the code (e.g. related to Python 3 conversion) were also
categorized as 'invalid clone URI'. And thus, the tests that test an
actually invalid URI would pass, even though there was a bug.

Now, things have been refactored so it only is relevant to catch
InvalidCloneUriException. Any other exception will now yield a 500 Internal
Server Error, as expected.

utils: let is_valid_repo_uri report URLError from ._check_url as InvalidCloneUriException

Better encapsulation of internals.

utils: don't raise NotImplementedError in is_valid_repo_uri when the user provides an "invalid" URL

We prefer to reserve NotImplementedError for programming errors that allow
access to unsupported corner cases in code.

validators: introduce InvalidCloneUriException instead of throwing bare Exceptions for invalid clone URIs

When adding a new repository with a remote clone URI, the URI will be validated
in some way. In case of errors, it would raise 'Exception' to report invalid
URLs. Code calling the validation would thus have to catch 'Exception', which
means that _any_ exception would cause the URI to be found invalid.

Instead, create a special exception type intended to be used for all exceptions
we know can occur during normal URL validation.

cleanup: improve exception logging from validators

There were no point in logging the whole raw exception with stacktrace - it is
more relevant to log something helpful

Catching of Exception is however bad.

Also, it seems odd what ValidPerms is doing with "update permissions" in the
validator.

vcs: fix repo creation with a remote clone uri under Python 3

When trying to add a new repository based on a remote clone, it fails in
Python 3 as follows, for git:

ERROR kallithea.model.validators:validators.py:413 URL validation failed
Traceback (most recent call last):
File ".../kallithea/model/validators.py", line 411, in _validate_python
is_valid_repo_uri(repo_type, url, make_ui())
File ".../kallithea/lib/utils.py", line 250, in is_valid_repo_uri
GitRepository._check_url(url)
File ".../kallithea/lib/vcs/backends/git/repository.py", line 174, in _check_url
if not test_uri.endswith('info/refs'):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

and for hg, the other way around:

ERROR kallithea.model.validators:validators.py:413 URL validation failed
Traceback (most recent call last):
File ".../kallithea/model/validators.py", line 411, in _validate_python
is_valid_repo_uri(repo_type, url, make_ui())
File ".../kallithea/lib/utils.py", line 233, in is_valid_repo_uri
MercurialRepository._check_url(url, ui)
File ".../kallithea/lib/vcs/backends/hg/repository.py", line 297, in _check_url
if os.path.isdir(url) or url.startswith(b'file:'):
TypeError: startswith first arg must be str or a tuple of str, not bytes

In both cases, the code seems to actually expect a bytes url.

cli: fix '--full' option of index-create

The option '-f' or '--full' is actually a flag, but the current declaration
made click expect a value as argument.

Instead, follow the click convention to allow --full and --no-full, to
indicate it is a boolean flag, as we do in other places.

cli: fix argv bug just introduced in 2ce0bc6e85f1

Reported by mjacob.

celery: fix send_email to work with JSON encoding (Issue #363)

Long time ago, c935bcaf7086 introduced an optional User object parameter to the
send_email task and used the computed full_name_or_username property. Due to the
magic of pickle, that also worked when using Celery to run the task async.

Now, Celery 4 changed the default encoding from Pickle to JSON, which we
anticipated in e539db6cc0da. That broke send_email in some cases, for example
when a user comments on another user's changeset.

Fixed by passing the "From" name as string instead of passing the whole User
object.

Thanks to vyom for reporting.

js: fix missing var declaration of AJAX_COMMENT_URL and AJAX_COMMENT_DELETE_URL (Issue #362)

Tech debt stroke back when fb9550946c26 cleaned things up.

Eslint wearnings for these variables had incorrectly been manually ignored.

front-end: add eslint-plugin-html as dependency as introduced in .eslintrc.js in 4d36432bf705

Usage example:
hg up -cr.
sed -i -e 's/\${[^{}]*\({[^{}]*}[^{}]*\)*}/""/g' -e 's/%\(if\|else\|endif\|for\|endfor\)\>.*//g' -e 's/##.*//g' $(hg loc 'kallithea/templates/**.html')
vim kallithea/templates/pullrequests/pullrequest.html +139 # blank out the multi line 'var url = ${}'
( cd kallithea/front-end/ && node_modules/.bin/eslint $(hg loc 'kallithea/templates/**.html')) | tee l
hg up -Cr.

( sed -n 's/^function \([^(]*\).*/\1/gp' kallithea/public/js/base.js; sed -n 's/.* var \([^ ]*\) =.*/\1/p' kallithea/templates/base/root.html; echo pyroutes; echo CodeMirror; echo Select2; echo BranchRenderer;)|while read x; do echo $x; sed -i "/error .$x. is not defined /d" l; done; cat l

front-end: move eslint to dev section and update package-lock.json

Eslint is only a dev tool.

auth: also use safe password hashing on Windows using bcrypt

For unknown reasons, Kallithea used different hashing algorithms on Windows and
Posix. Perhaps because problems with bcrypt on Windows in the past. That should
no longer be an issue, and it doesn't make sense to have different security
properties on the platforms.

While changing to bcrypt everywhere, also remain backwards compatible and
accept existing passwords hashed with sha256 - both on Windows (where it used
to be used), and elsewhere (in case a system has been migrated from Windows to
Unix).

cli: pass local argv to argparser instead of letting it use the global sys.argv

Fix pylint warning: "fix W0613(unused-argument) in argparser: Unused argument 'argv".

cleanup: minimize use of lambda expressions - we have 'def' for that purpose

Fix some flake8 warnings "E731 do not assign a lambda expression, use a def".

celery: upgrade to Celery 4

Celery 3 doesn't support Python 3.7 or later. This upgrade is thus essential
for full Python 3 support. But note that
https://docs.celeryproject.org/en/4.4.0/faq.html#does-celery-support-windows
says "No".

The names of config settings changed in Celery 3 to 4, as described on
https://docs.celeryproject.org/en/3.0/whatsnew-4.0.html#lowercase-setting-names .

Celery 4 config settings can now be specified in Kallithea .ini files by prefixing
with `celery.` - for example as `celery.broker_url`.

Remain backwards compatible for the usual settings, and map old names to the
new names.

celery: move initialization of global kallithea.CELERY_EAGER to the place where we read celery configuration

Prepare for backwards compatibility after the name change with Celery 4.

celery: log an error for some deprecated config settings

dburi was dropped from templates in d34a59bfcebb1 and is no longer supported
in Celery 4.

celery.send.task.error.emails will no longer be supported in Celery 4 and was
just removed from the .ini template .

The "serialier" typo was fixed in 8169770a4f2c. It might bite people if they
keep using the wrong spelling.

celery: drop config settings not supported in Celery 4

CELERY_SEND_TASK_ERROR_EMAILS has been removed, according to
https://docs.celeryproject.org/en/3.0/whatsnew-4.0.html#features-removed-for-simplicity :
"Tasks no longer sends error emails. This also removes support for
app.mail_admins, and any functionality related to sending emails."

The "ADMINS" prefix might be a left-over from the long removed
celery.conf.ADMINS - last seen on
https://docs.celeryproject.org/en/2.1-archived/reference/celery.conf.html#celery.conf.ADMINS .

The "ROUTES" prefix might refer to "CELERY_ROUTES" ... but it doesn't take a
simple string list anyway, so there is no point in treating it as a list value.

celery: change ini template to use sqlite for results

'amqp://' might be good - also for results, but seems to need additional
non-trivial setup. And according to
https://docs.celeryproject.org/en/3.0/whatsnew-4.0.html#features-removed-for-lack-of-funding
it is deprecated.

Kallithea only uses Celery results when repos are created or forked and user
browsers are reloading pages to poll for completion. amqp seems like
unnecessary complexity for that use case.

Sqlite does however seem like a minimal but fine solution for the Kallithea use
case in most setups.

celery: set default config values in code and remove them from the generated .ini

It is hard to imagine any reason the user should change celery.imports . And if
it ever should change, we want it controlled in code - not left stale in user
controlled config files.

Everybody sould just use .json and there is no reason anybody should specify
that in the .ini ... and it will be the default in Celery 4.

config-create: show options used for generating .ini, and list what alternative values are supported

Give users a better hint that they should specify web server when generating .ini .

eslint: install eslint with front-end so it can be run manually

eslint can be run manually on the main .js files with:

./node_modules/.bin/eslint kallithea/public/js/base.js kallithea/public/js/codemirror_loadmode.js kallithea/public/js/graph.js

- but it is quite noisy. Some problems do however stand out as relevant to fix.

Script sections in HTML files can also be checked after removing mako markup:

hg up -cr.
sed -i -e 's/\${[^{}]*\({[^{}]*}[^{}]*\)*}/""/g' -e 's/%\(if\|else\|endif\|for\|endfor\)\>.*//g' -e 's/##.*//g' $(hg loc 'kallithea/templates/**.html')
vim kallithea/templates/pullrequests/pullrequest.html +139 # blank out the multi line 'var url = ${}'
./node_modules/.bin/eslint $(hg loc 'kallithea/templates/**.html')
hg up -Cr.

- but that is even more noisy.

The noise is mainly due to eslint not knowing that everything runs together,
with kallithea/templates/base/root.html defining global variables,
kallithea/public/js/base.js using these and defining functions, which then is
used "everywhere". There might be solutions to that - this is a starting point.

js: use strict ... and fix the problems it points out

"use strict" gives stricter checks, both statically and at runtime. The
strictness tightens up the code and prevents some kinds of problems.

The <script> tag addition might not be pretty, but has consistently been added
with:

sed -i 's,<script>$,&'"'"'use strict'"'"';,g' `hg loc '*.html'`

js: omit redundant type="text/javascript"

We already used plain '<script>' in several places.

https://www.w3.org/TR/html52/semantics-scripting.html#element-attrdef-script-type :
"Authors should omit the attribute, instead of redundantly giving a JavaScript
MIME type."

repo: fix sorting in repo lists (Issue #360)

Fix breakage I introduced in 173612a900ef when modifiying to add an early
column and thus shifting the indices of columns used for sorting.

celery: update ini template to use json for tasks and results

Json seems to be more reliable than pickle, and seems to be the future.

celery: fix logging from inside tasks

Celery will hijack the global logging configuration and disable the usual
Kallithea logging (unless CELERYD_HIJACK_ROOT_LOGGER is set False).

Fixed partially by using the special celery loggers inside the tasks. Logging
from modules will still not work.

celery: use celery directly instead of leaky abstraction in celerypylons

Things start making more sense when we remove unnecessary complexity ...

celery: use the proper configured global app for scheduling and retrieving tasks

193138922d56 broke celery, due to magic dependencies on initialization and
setting global configuration at import time.

Instead, always use the correctly configured global Celery app, both when
creating tasks and checking result status.

This has been tested to work on Python 3.6 - for example for sending mails and
forking repos.

Celery has however been found to not work on Python 3.7, due to Celery 3.x
using the new reserved keyword 'async'.

celery: refactor initialization - replace global CELERY_ON flag with CELERY_APP with the actual celery app that it indicates

Prepare for fixing how 193138922d56 broke celery due to magic dependencies on
initialization of global state at import time.

ui: show toggleable "Follow" status in repo groups' repo list

It makes sense to show Follow status next to repo names in the repo list, and
it is a meaningful and efficient bulk operation to toggle Follow status there.

Clicking on the (Un)Follow 'heart' will toggle the caller's follow status for
that repo.

The repo model already has layering violations - expand on them to compute the
follow status of the current user.

(Changeset was cherry picked and modified by Mads Kiilerich.)

cleanup: run pyflakes as a part of scripts/run-all-cleanup

pyflakes has no usable configuration options, so create a small wrapper script.
Instead of having two wrapper scripts (with one being almost nothing and the
other containing configuration), just keep it simple and use one combined.

ini: drop deprecated CELERY_RESULT_DBURI

https://docs.celeryproject.org/en/3.1/configuration.html#celery-result-dburi says:

This setting is no longer used as it’s now possible to specify the database
URL directly in the CELERY_RESULT_BACKEND setting.

ui: fix page navigation/scrolling when toggling "Follow" status

The onclick handler was a void function and thus not cancelling the default
handling of following the 'href'.

The called toggleFollowingRepo function was already returning false, so the
onClick handler just needs to return the result of the called function.

ini: fix spelling of the CELERY_TASK_SERIALIZER

It has "always" been wrong, and we have thus been using the default of "pickle"
- not "json" as we thought.

I doubt this change has any immediate visible impact. I guess it only means
that it will use json for results instead of pickle. That might be more stable
and debuggable.

Note: celery_config will uppercase the config settings and replace '.' with
'_', so 'celery.result.serializer' turns into 'CELERY_TASK_SERIALIZER'.

CELERY_TASK_SERIALIZER is described on
https://docs.celeryproject.org/en/3.1/configuration.html#celery-result-serializer
https://docs.celeryproject.org/en/3.1/userguide/calling.html#serializers

mails: make error reporting by mail work with secure mail servers

Even with Kallithea mails working, TurboGears / backlash error reporting would
fail like:

Error while reporting exception with <backlash.tracing.reporters.mail.EmailReporter object at 0x7f8f986f8710>
Traceback (most recent call last):
File ".../env/lib/python3.7/site-packages/backlash/tracing/reporters/mail.py", line 49, in report
result = server.sendmail(self.from_address, self.error_email, msg.as_string())
File "/usr/lib64/python3.7/smtplib.py", line 867, in sendmail
raise SMTPSenderRefused(code, resp, from_addr)
smtplib.SMTPSenderRefused: (530, b'5.7.0 Must issue a STARTTLS command first.', 'kallithea@example.com')

Extend the .ini template to forward all the mail settings described on
https://turbogears.readthedocs.io/en/latest/reference/config-options.html#error-reporting

follow: Fix args for $.post() call in toggleFollowingRepo()

Before, the resulting query string was _literally_ like:

?follows_repository_id=xxx&amp;_session_csrf_secret_token=yyy

The server side stack ended up parsing it as:

>>> urllib.parse.parse_qsl("follows_repository_id=xxx&amp;_session_csrf_secret_token=yyy", keep_blank_values=True)
[('follows_repository_id', 'xxx'), ('amp', ''), ('_session_csrf_secret_token', 'yyy')]

because the HTML encoding of & isn't relevant here and it thus looks like a
"matrix URL" (as drafted on https://www.w3.org/DesignIssues/MatrixURIs.html )
with the following values:

param: follows_repository_id = xxx
param: amp =
matrix: _session_csrf_secret_token = yyy

It thus ended up with the right values (if ignoring 'amp') and it thus worked
anyway.

Instead, clean it up and just pass a dict to jQuery.post as intended.

ui: use fork icon for the fork option (Issue #356)

Reported with patch by Ross Thomas.

setup: bump version numbers to current max

This will most likely give best py3 support, and we need thorough testing
anyway.

Based on 'pip freeze' before/after:

sed -i '/>=[^#]*$/s/, < [0-9.]\+//g' setup.py dev_requirements.txt

vcs: drop _fix_path

The reasoning in hg _fix_path is no longer correct.

We only need it for stripping the trailing '/' ... but we would rather be
explicit about that. (But it is also questionable if we actually want it at all
- we should just pass the right data and fail on wrong data.)

i18n: updated translation for Polish

Currently translated at 56.5% (614 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 100.0% (1087 of 1087 strings)

py3: officially support Python 3

All tests pass and no known regressions.

py3: work around incompatibility between pytest, py3 inspect, and tg

Work around an issue that has been reported on
https://github.com/TurboGears/tg2/issues/118 :

.../site-packages/_pytest/doctest.py:381: in _mock_aware_unwrap
return real_unwrap(obj, stop=_is_mocked)
/usr/lib64/python3.7/inspect.py:511: in unwrap
while _is_wrapper(func):
/usr/lib64/python3.7/inspect.py:505: in _is_wrapper
return hasattr(f, '__wrapped__') and not stop(f)
.../site-packages/tg/support/objectproxy.py:19: in __getattr__
return getattr(self._current_obj(), attr)
.../site-packages/tg/request_local.py:240: in _current_obj
return getattr(context, self.name)
.../site-packages/tg/support/objectproxy.py:19: in __getattr__
return getattr(self._current_obj(), attr)
.../site-packages/tg/support/registry.py:72: in _current_obj
'thread' % self.____name__)
E TypeError: No object (name: context) has been registered for this thread

pytest's doctest support is (in _mock_aware_unwrap) using py3 inspect.

Inside inspect, _is_wrapper will do an innocent looking:
hasattr(f, '__wrapped__')

But if the code under test has un (unused) import of a tg context (such as
tg.request), it is no longer so innocent. tg will throw:
TypeError: No object (name: context) has been registered for this thread
(which in py2 would have caught by hasattr, but not in py3.)

pytest will thus fail already in the "collecting ..." phase.

To work around that, use the hack of pushing a tg context in the top level
pytest_configure.

tests: update test_filenode_path for py3

Unicode is now handled very differently. There are no encoded unicode str, and
the test barely makes sense any more.

tests: accept py3 output in test_list_valued_setting_update

The old behaviour seems completely broken and useless and thus evidently unused
... but the new result seems slightly better.

py3: fix non-ASCII URLs - decode unicode correctly before passing them to controllers as unicode strings

This is needed for supporting localized repo path names in the path of URLs.

Some references:

https://www.python.org/dev/peps/pep-0333/#unicode-issues
https://bugs.python.org/issue16679
http://lucumr.pocoo.org/2010/5/25/wsgi-on-python-3/
https://bugs.launchpad.net/pecan/+bug/1451842
https://github.com/tipabu/eventlet/commit/a5a7751b013fe99b6d30acbca79e819770e9ae5d

py3: update safe_* functions for how unicode pretty much has been renamed to str

safe_str and safe_unicode is thus the same thing.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: use inspect.getfullargspec instead of deprecated inspect.getargspec

Avoid some pytest warnings.

py3: fix kallithea-cli ini parsing after ConfigParser got strict=True

ConfigParser in py3 defaults to strict=True and would thus reject our ssh
logging hack of renaming config sections ... which cause duplicate section
names.

Fortunately, fileConfig now also allows passing a ConfigParser, and we can
avoid using io.StringIO .

py3: update ssh for base64.b64decode raising binascii.Error instead of TypeError

A command like:
python -c 'import base64; base64.b64decode("QQ")'
would fail in Python2 with:
TypeError: Incorrect padding
but in python3:
binascii.Error: Incorrect padding

py3: use html.escape instead of cgi.escape

DeprecationWarning: cgi.escape is deprecated, use html.escape instead

py3: migrate from urllib2 to urllib

Based on 2to3 urllib, but with unnecessary imports removed.

py3: support new stdlib module names configparser, urllib and http lib

From "2to3 -f imports".