cleanup: fix db.py "import *"

pyflakes rightfully complains about wildcard imports. But we like to keep the
short names, so let's just be explicit about which names we import.

py3: fix error in template from introducing safe_unicode in 9203621cae03

Add missing test coverage of actual edit form.

validator: fix ASCII password check to verify if it can be *encoded* in ascii

In Python 2, unicode strings have a .decode method (which really doesn't make
sense). Python 3 has more strict typing by design, and unicode strings don't
have a .decode method.

A Unicode string "is ASCII" if it can be encoded as ASCII. The check should
thus *encode* to ASCII - not decode.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

py3: drop __unicode__ ... and generally move towards just providing a good __repr__

We should never show str() (or similar) to end users.

py3: drop .keys when we don't need them

In python 3 they will be iterators and mostly useless ... but they are already
mostly redundant in py2.

controllers: use unicode for internal rendering of template snippets

Like 8e2313be042d.

gist: make it a bit more clear how gist_access_id is used ... and how it is different from gist_id

A gist has a gist_access_id which gives access to it. For private Gists, it is
a multi-letter secure random string.

gist_id is the primary key in the database and thus an automatically
incrementing integer. It is also used as the not-so-secret gist_access_id for
public gists.

This gets rid of one odd safe_unicode applied to an int.

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

lib: refactor _get_access_path as get_path_info

We will need it later when it gets more tricky to get the path from environ ...

lib: avoid comparing ordering with None

Fails in py3.

lib: mktime needs a tuple - not a list

Py3 is more strict.

lib: base64 followup from 82b1eaec25f5

Needed for py3.

lib: avoid import string.replace

Odd way to do it ... and gone in py3.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

vcs: make hg get_changesets compatible with py3

If start was 0, we would end up with start_pos None and getting a compare with
None which would fail in py3.

Instead, make sure start_pos is None iff start is None.

Also, make the actual check more robust.

vcs: change check_integrity to work on paths

Py3 failed because nodes can't be hashed.

vcs: refactor and simplify _get_revision

This will be more thorough checking that revisions actually exist, and will in
some cases give better error messages.

But mainly, prepare for py3 where this was easier to make work correctly.

vcs: refactor run_git_command to just return stdout as unicode string

This will be convenient for py3.

vcs: change get_diff to use _run_git_command directly

This will allow run_git_command to return strings while get_diff still return
raw bytes.

lib: let get_git_version invoke git as all other commands do, without special options

There is no need for _bare or _safe. It is fine to have '-c
core.quotepath=false' before '--version', and it is perfectly fine to get a
RepositoryError if things go terribly wrong.

vcs: use more correct git revision identifiers

It will have the right types and make a difference for py3.

vcs: replace __getslice__ with slice handling in __getitem__

Needed for py3.

vcs: fix __eq__ to return false if types are different

Follow up to ae12fabba699.

vcs: drop Repository.name_unicode

It wouldn't make much sense in py3 ... and is barely used.

vcs: replace "has_key" with "in" in decorator skip criteria

Needed for py3.

tests: fix ordering of get_inline_comments output

Py3 would give different ordering.

tests: fix test_files.py json comparison to load json to struct so it is independent of serialization order

Py3 would fail the test because different ordering.

tests: sort groups in test_enforce_groups check to make it independent of ordering

Py3 would give another ordering.

tests: fix ordering of options of "ret_type must be one of ..." returned from api

Py3 would fail becauase it use another ordering.

tests: avoid unicode u'' repr in test_diff_markup output

It will go away with py3 anyway.

tests: refactor test_redirection_to_login_form_preserves_get_args to test more correctly

Preparing for py3.

lib: introduce string conversion functions for ASCII without further encoding concerns

Avoid the trial-and-error and vagueness of the "safe" functions.

This should replace some use of safe_unicode and safe_str. It will mostly be a
noop in py2 but will be crucial in py3.

lib: establish py3 compatible strategy for string handling: introducing safe_bytes and deprecating safe_str

The meaning of safe_str will change when moving to py3. All use of safe_str is
thus tech debt that we have to chop off, mostly by moving to either
safe_unicode or safe_bytes ... or dropping because we know what we are doing
and rely on the improved type safety in py3.

vcs: drop unused _diff_name_status

It popped up as something to do for py3. Don't waste time on it.

lib: some cleanup of utils.py imports

Utils really shouldn't rely on model ...

hg: bump minimum version to 5.1

We will soon move to Python 3 which only will support 5.1 or later.

Remove old hacks and tech debt.

Also avoids future warning:
DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()

py3: only use safe_str for string conversion - not for arbitrary __str__ invocation

Usually, we actually just want unicode string, and it is better to just unicode().

lib: only maintain one copy of safe_str / safe_unicode

The standalone-ish nature of vcs gets a bit in the way. It already depends on
some very generic Kallithea functionality. But for now, avoid code duplication,
and let Kallithea use vcs functionality instead of duplicating it.

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

feeds: replace webhelpers.feedgenerator with simple mako templating for rendering RSS/Atom XML

Most of the complexity in RSS libraries is in dynamically supporting all kinds
of attributes. For our use, we have a small static set of attributes we use,
and it is simpler to just use mako.

Also, webhelpers is dead, and the alternatives seem quite heavy.

repo: don't just report user name and email in one field - separate things properly

In the repo RSS feed, report author as
<author>name@example.com (User Name)</author>
instead of using
<dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">User Name &lt;name@example.com&gt;</dc:creator>

And in the ATOM feed with name and email separate:
<author>
<name>User Name</name>
<email>name@example.com</email>
</author>
Instead of
<name>User Name &lt;name@example.com&gt;</name>

journal: don't include email in author name - avoid double data

In journal RSS feed, report author as:
<author>name@example.com (User Name)</author>
instead of double email due to:
<author>name@example.com (User Name &lt;name@example.com&gt;)</author>

In the journal ATOM feed, report author as:
<author>
<name>user Name</name>
<email>test_admin@example.com</email>
</author>
instead of using double email due to:
<name>name@example.com (User Name &lt;name@example.com&gt;)</name>

i18n: updated translation for Russian

Currently translated at 100.0% (691 of 691 strings)

py3: use comprehensions and generators instead of filters - it is more explicit, and sometimes shorter

From 2to3 -f filter.

py3: cleanup map usage and avoid py3 ambiguity

Based on 2to3 -f map ... but replace map with something more explicit (unless
born and raised in a lisp world) (but sometimes slightly more verbose).

py3: prepare for types.NotImplementedType going away

From 2to3 types.

py3: use explicit octal literals

From 2to3 numliterals.

model: use unicode for internal rendering of repo snippets

Preparing for py3.

cleanup: use isinstance instead of comparing types

From 2to3 idioms.

tests: fix some odd source formatting / spacing around ','

From 2to3 -f ws_comma.

changeset: store hexified source hash in context so it is readily available in template

Avoid hex as "encoding" - it is not supported in py3.

Mercurial store source and transplant_source differently ... but that will now
be handled in the controller instead of in the template.

home: don't send _changeset_cache in repo_switcher_data result

It is unused, and it is problematic to serialize bytes in json ... and it is
especially obvious in py3.

lib: drop redundant json.dumps encoding=utf8

Not supported in py3 - json is by definition(?) always UTF-8.

app: drop finally handling in BaseVCSController.__call__

Our optional 'wrapper' middleware is logging response times more correctly
after b42ee1bdf082 - there is no point in logging the timing of the __call__ in
the main app.

Similarly, the session was removed too early. But TurboGears is already picking
up our db session and using DBSessionRemoverMiddleware to .remove() it ... and
at the right time. So just stop trying.

utils: drop make_ui parameter clear_session - this utility function should never mess with sessions

clear_session seems to have been based on the idea that utility functions that
happened to create a session also should clean up. But instead, we now
consistently take care of session removal at a high level.

app: let app instance creation remove its SA Session when done

The app instance is usually created early or on demand before serving the first
request, and the app instance might sit idle for a while before receiving the
first request. The app creation will touch the database, and we have to
consider the lifetime of the database session it is using. The cbf524e4c1a3
commit message gives some reasons we should remove the session when done.

But instead of handle it inside utility functions, we would rather handle
sessions at a high level as here.

home: drop disabled caching of repo_switcher_data

The caching condition has been False since ffd45b185016 . That makes sense -
caching as implemented would leak the repo list of other users without
considering permissions. The repo list is however only loaded on demand, and
caching is not that important.

routing: fix files_annotate_home annotate value to be compatible with Routes >= 2

The routing entry for files_annotate_home had annotate=True. That primarily
served to let the controller files.index differentiate files_annotate_home from
files_home and files_home_nopath . Anything true can work.

test_files.py is creating files_annotate_home URLs in an odd way: Instead of
explicitly specifying it is a files_annotate_home URL, it passes expected
controller parameters and expects routing to find a URL that would provide
these parameters. It thus also has to specify the otherwise "invisible"
annotate value. For Routes < 2, True works just fine. For Routes >= 2, it seems
to expect values that actually can be encoded in URLs.

Thus, instead of True, use '1'.

routing: drop default f_path for changelog_file_home

There is no point in creating a changelog_file_home URL without specifying
f_path. The default value of None is thus never any good. And especially, it is
unclear what a value of None really means.

routing: drop action='index' - that is already the default

These default actions were already left out in some places - now be consistent
about it.

ssh: fix parsing of ed25519 keys

When attempting to use ed25519 SSH keys, parse_pub_key() failed with:
SshKeyParseError: Incorrect SSH key - base64 part is not 'ssh-ed25519' as claimed but 'ssh-ed25519'

The problem was the hardcoding of the string length of the key type -- 7 or
'\x07' -- which fits ssh-rsa and ssh-dss but not ssh-ed25519.

(Fix was simplified by Mads Kiilerich.)

ssh: add doctest showing incorrect handling of ed25519 keys

This doctest just demonstrates that ed25519 keys are not treated correctly
by parse_pub_key(). A subsequent commit will actually fix it.

scripts/make-release: install ldap and pam to fix isort instabilities

isort (triggered by scripts/whitespacecleanup.sh) needs to know which
modules are local and which are not, to separate them with a newline. If a
module cannot be found, it will be treated as local, apparently.

When ldap is not installed in the current environment, a difference was
created by isort in kallithea/bin/ldap_sync.py.

bin/ldap_sync: revert commit 04dee6fdfdff

Commit 04dee6fdfdff fixed an apparent problem detected by isort, but as Mads
Kiilerich pointed out, it was caused by an incomplete virtualenv that did
not include 'python-ldap'. As a result, isort would not consider 'ldap' as a
standard module and treated it as a local module.

A subsequent commit will fix the 'make-release' script to install all
expected dependencies.