pullrequests: introduce limit to stop displaying additional changes
We've noticed some scalability issues when many descendants exist for the changesets in a pull request.
To resolve this issue, we instead do not display any additional changes at all if the amount of additional changes is larger than a configured limit.
(This occurred because we were merging a lot of heads in the repository.)
Throwing away some of the changesets would also keep the total amount more manageable, but this would result in an (for the user) unpredictable subset of changesets being shown.
This could be extended further with "too long to be shown - click here to show", but that was quite a bit of additional work and did not cover our use case of not allowing the display at all in case of too many additional changes.
After renaming a group, it would iterate over all the contained groups and repos and update their full path while logging the update from the/old/path to the the/new/path. Doing that, it would also visit the already renamed top level group, but since the full path of that one already had been updated, it would log it as renaming from the/new/path to the/new/path.
Fixed by logging when renaming in the first place, and skipping the top level repo group while iterating.
To avoid redundant logging, only log (and rename) if the name or parent actually change.
setup: use old importlib_metadata version to fix kombu failing on python < 3.8
Many libraries use the importlib_metadata library as fallback when running on Python versions older than 3.8 . For example setuptools when easy_install is used for install the Kallithea console_scripts entrypoints in the bin folder. The dependencies on importlib_metadata were indirect and without constrains on version number.
The problem is that Celery uses Kombu, which (on Python < 3.8) uses importlib_metadata in a way that is incompatible with importlib_metadata > 5.
Most obvious, building docs failed as: Running Sphinx v5.1.1
Configuration error: There is a programmable error in your configuration file:
Traceback (most recent call last): File ".../kallithea/venv/lib64/python3.7/site-packages/sphinx/config.py", line 347, in eval_config_file exec(code, namespace) File ".../kallithea/docs/conf.py", line 17, in <module> import kallithea File ".../kallithea/kallithea/__init__.py", line 45, in <module> CELERY_APP = celery.Celery() # needed at import time but is lazy and can be configured later File ".../kallithea/venv/lib64/python3.7/site-packages/celery/local.py", line 492, in __getattr__ [name]) File ".../kallithea/venv/lib64/python3.7/site-packages/celery/app/__init__.py", line 2, in <module> from celery import _state File ".../kallithea/venv/lib64/python3.7/site-packages/celery/_state.py", line 15, in <module> from celery.utils.threads import LocalStack File ".../kallithea/venv/lib64/python3.7/site-packages/celery/utils/__init__.py", line 16, in <module> from .nodenames import nodename, nodesplit, worker_direct File ".../kallithea/venv/lib64/python3.7/site-packages/celery/utils/nodenames.py", line 6, in <module> from kombu.entity import Exchange, Queue File ".../kallithea/venv/lib64/python3.7/site-packages/kombu/entity.py", line 7, in <module> from .serialization import prepare_accept_content File ".../kallithea/venv/lib64/python3.7/site-packages/kombu/serialization.py", line 440, in <module> for ep, args in entrypoints('kombu.serializers'): # pragma: no cover File ".../kallithea/venv/lib64/python3.7/site-packages/kombu/utils/compat.py", line 82, in entrypoints for ep in importlib_metadata.entry_points().get(namespace, []) AttributeError: 'EntryPoints' object has no attribute 'get'
That made readthedocs builds fail, when it in the default web configuration used Python 3.7 .
Fixed by introducing an explicit dependency on importlib_metadata < 5.
The repo group owner concept was only partially implemented. Owners were shown in the repo group listing, but couldn't be changed. Users owning repo groups couldn't be deleted, with no other solution than deleting owned repo groups.
This also fixes the existing broken update_repo_group API, which tried to use unimplemented functionality.
repo: introduce enable_downloads and enable_statistics when creating repos
These booleans were not shown in the normal repo creation form, so the form validation applied the "default" values of False. These values were however not used by the model when creating repos - it just unconditionally used the real global defaults.
The API already exposed some of this, but it wasn't implemented.
The web form for creating repos lacked these fields, but it was present in the repo edit form. Just make these fields mandatory. There will thus not be any defaults to apply in the model for creating repos.
Sphinx 3.0.4 doesn't version its dependencies correctly, and it fails with latest jinja 3.1.2: ImportError: cannot import name 'environmentfilter' from 'jinja2' (.../site-packages/jinja2/__init__.py)
Latest Sphinx version supports latest Jinja version and seems to render the documentation correctly.
070b8c39736f accidentally introduced a wrong assumption that url_scheme_variable is a bool. Fix to only check whether it has been set to something non-empty.
middleware: use config consistently in https_fixup
070b8c39736f did for unknown reasons introduce a use of the global kallithea.CONFIG . Instead, consistently use the application config object that has been passed.
Clicking "Add new" on /_admin/user_groups/1/edit/perms would flicker the input fields, and then show "User group permissions updated".
That turns out to be because HTML button tags default to 'submit' inside forms, and in this place (but not for user or repo group permissions) we didn't specify the button type as 'button'.
Deletion of a repository group that has a parent group (i.e. is not at the root of the repository group tree) failed as follows:
Traceback (most recent call last): [...] File ".../lib/python3.9/site-packages/tg/configurator/components/dispatch.py", line 114, in _call_controller return controller(*remainder, **params) File "<decorator-gen-5>", line 2, in delete
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/auth.py", line 572, in __wrapper return func(*fargs, **fkwargs) File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/controllers/admin/repo_groups.py", line 271, in delete if gr.parent_group: File ".../lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__ return self.impl.get(instance_state(instance), dict_) File ".../lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 730, in get value = self.callable_(state, passive) File ".../lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 717, in _load_for_state raise orm_exc.DetachedInstanceError( sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <RepoGroup at 0x7f1f2664f4c0> is not bound to a Session; lazy load operation of attribute 'parent_group' cannot proceed (Background on this error at: http://sqlalche.me/e/13/bhk3)
In the reference 'gr.parent_group', 'gr' is an SQLAlchemy object referring to the group being deleted, and 'gr.parent_group' is a lazy reference to its parent group. The 'lazy' means that the parent group object is not loaded automatically when 'gr' is assigned, but instead will be loaded on-the-fly when the parent group is actually accessed. See [1] and [2] for more information.
The problem was that the lazy 'parent_group' attribute was accessed _after_ deleting the database object it was part of.
Fix this by obtaining a handle to the parent group _before_ deleting the subgroup.
controllers: don’t pass start=0 to BaseRepository.get_changesets()
MercurialRepository.get_changesets() can fail if passing start=0 if the revision 0 is not in self.revisions. That can happen if revision 0 is not in the visible subset of the revisions in the repository. Before Kallithea changeset 7c43e15fb8bc7a73f17f577e59a4698589b6809d, it was working by chance because start=0 was treated like start=None in the relevant places (GitRepository.get_changesets still does that).
The intention of passing start=0 was seemingly to not limit the start. Therefore passing start=None (or nothing, as it’s the default value) should be correct.
I got the following traceback before this change:
Traceback (most recent call last): File "~/vcs/kallithea/kallithea/controllers/changelog.py", line 117, in index collection = c.db_repo_scm_instance.get_changesets(start=0, end=revision, File "~/vcs/kallithea/kallithea/lib/vcs/backends/hg/repository.py", line 529, in get_changesets start_pos = None if start is None else self.revisions.index(start_raw_id) ValueError: '4257f758b3eaacaebb6956d1aefc019afab956b8' is not in list
Running "kallithea-cli front-end-build" with npm 7.21.1 gave:
npm WARN old lockfile The package-lock.json file was created with an old version of npm, npm WARN old lockfile so supplemental metadata must be fetched from the registry. npm WARN old lockfile npm WARN old lockfile This is a one-time fix-up, please be patient...
files: fix raw download of repo files with names with unicode points above 256 in name
Raw download had apparently only been tested with non-ascii characters that were latin1. That was apparently a (too) simple case that worked without crashing.
Files with unicode code points above 256 in their name would fail to download, when Waitress failed like this, trying to get a real byte string by encoding WSGI headers to latin1: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 84-85: ordinal not in range(256)
HTTP headers are of course byte strings on the network, but Python3 WSGI does unfortunately neither expose it as bytes nor as unicode strings to be encoded as utf-8. Instead, it uses unicode strings with byte values encoded as code points 0-255. That is achieved by decoding the utf-8 encoded bytes as latin1.
For raw downloads, the recommended download filename is provided in the Content-Disposition header. The problem is that it was provided as a real unicode string.
Fixed by applying the "proper" latin1-decoding of a utf8-encoding.
api: fix repo group permission check for repo creation
hg.create.repository is only controlling whether all users have write access at top level. For other repo locations, check for write permission to the repo group.
Note: This also covers "repo creation" by forking or by moving another repo.
front-end: use 'bin' path for node commands instead of '.bin'
license-checker is using relative paths for importing other modules - that worked fine when .bin/license-checker was a symlink, but not on filesystems without symlinks support:
git: make sure _check_url only accept the protocols accepted by is_valid_repo_uri
Avoid unnecessary flexibility, ambiguity, and complexity.
The file protocol was never used. But when cloning existing managed repos, is_valid_repo_url would be skipped and _check_url would be called with absolute paths.
repo_groups: fix select of parent group when adding repo group
h.select was passed a list of repo groups where group_id was integer, but parent_group in the request was a string - thus no match.
Do as in repos controller create_repository (and in error handling): leave it to htmlfill to patch up the generated HTML using defaults ... but make sure we always have a default.
repos: extra HTML escaping of repo and repo group names shown in DataTables
These names will already have been "slugged" and can thus not contain anything that can be used for any attack. But let's be explicitly safe and escape them anyway.
raw_name without escaping would cause XSS *if* it was possible to create unsafe repo names.
just_name must be escaped in order to make search work correctly - for example if searching for '<' ... *if* it was possible for names to contain that.
It is checked earlier that git_command is one of two string constants, and with py3, things are much simpler and we don't have to consider string coersion.