Testing fontification, indentation, and buffer manipulation

classic Classic list List threaded Threaded
46 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Testing fontification, indentation, and buffer manipulation

Daniele Nicolodi
Hello,

I am hacking on beancount-mode (a mode to edit Beancount ledger files)
and I would like to write some automated tests to check the
functionality of the minor mode.  I found ERT, but it seems that it does
not offer any facility to easily test fontification, indentation, or
buffer manipulation in general.

Is there any facility that would help in writing such tests?

Thank you.

Cheers,
Dan

Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Yuri Khan
On Tue, Jan 15, 2019 at 10:52 PM Daniele Nicolodi <[hidden email]> wrote:

> I am hacking on beancount-mode (a mode to edit Beancount ledger files)
> and I would like to write some automated tests to check the
> functionality of the minor mode.  I found ERT, but it seems that it does
> not offer any facility to easily test fontification, indentation, or
> buffer manipulation in general.
>
> Is there any facility that would help in writing such tests?

You might want to look at ledger-mode tests. Its approach to
fontification testing is:

* Create a temporary buffer.
* Put test text into it.
* Harvest it back with fontification properties into a data structure.
* Test that this structure is equal to the golden output.

https://github.com/ledger/ledger-mode/blob/master/test/test-helper.el#L149
https://github.com/ledger/ledger-mode/blob/master/test/fontify-test.el

Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Helmut Eller-2
In reply to this post by Daniele Nicolodi
On Tue, Jan 15 2019, Daniele Nicolodi wrote:

> I am hacking on beancount-mode (a mode to edit Beancount ledger files)
> and I would like to write some automated tests to check the
> functionality of the minor mode.  I found ERT, but it seems that it does
> not offer any facility to easily test fontification, indentation, or
> buffer manipulation in general.
>
> Is there any facility that would help in writing such tests?

Something that I found useful was using somewhat exotic characters to
mark positions.  E.g.

  (forth-assert-face ": →frob ;" font-lock-function-name-face)

tests that the face after the → is font-lock-function-name-face.

Or

  (forth-assert-forward-sexp " ¹if drop exit else 1+ then² bar ")

tests that if forward-sexp is called at the position marked with ¹ it
will move to ².

Helmut


Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Daniele Nicolodi
In reply to this post by Yuri Khan
On 15/01/2019 11:45, Yuri Khan wrote:

> On Tue, Jan 15, 2019 at 10:52 PM Daniele Nicolodi <[hidden email]> wrote:
>
>> I am hacking on beancount-mode (a mode to edit Beancount ledger files)
>> and I would like to write some automated tests to check the
>> functionality of the minor mode.  I found ERT, but it seems that it does
>> not offer any facility to easily test fontification, indentation, or
>> buffer manipulation in general.
>>
>> Is there any facility that would help in writing such tests?
>
> You might want to look at ledger-mode tests. Its approach to
> fontification testing is:
>
> * Create a temporary buffer.
> * Put test text into it.
> * Harvest it back with fontification properties into a data structure.
> * Test that this structure is equal to the golden output.
>
> https://github.com/ledger/ledger-mode/blob/master/test/test-helper.el#L149
> https://github.com/ledger/ledger-mode/blob/master/test/fontify-test.el

Thanks Yuri, this is very helpful.

Cheers,
Dan



Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Jostein Kjønigsen
Hey Daniele.

In several of the (third-party) major-modes I author or maintain, I use the library "assess" available on MELPA for testing fontification. See: https://melpa.org/#/assess

For testing indentation, I've also created reference-documents and completely reindented them in a test and then  compared the results. This provides some basic (but not complete) coverage, because (depending on the mode) interactive usage may give different results.

An example. Consider the following line of c/java/javascript/php/c#-code:

    if (someBool)

The correct indentation for the following line could be:

- the same as the preceeding line if the next line contains an opening bracket ({)
- nested one level deeper if it contains a direct expression.

So when pressing enter... Should you indent immediately? And should you then later on correct the indentation if a user creates a block-start marker { ?

I'm not sure what the right answer is, but it is an example of something which will differ in a interactive use-case vs reflowing a completed, existing document.

If your major-mode has cases like this, you will have to write more elaborate tests to ensure these too are handled correctly.

That said: I'm not sure if this is the recommended or canonical approach, but combining these techniques has worked for me in my projects.

--
Regards
Jostein Kjønigsen



On Sat, Jan 19, 2019, at 4:07 PM, Daniele Nicolodi wrote:
On 15/01/2019 11:45, Yuri Khan wrote:
On Tue, Jan 15, 2019 at 10:52 PM Daniele Nicolodi <[hidden email]> wrote:

I am hacking on beancount-mode (a mode to edit Beancount ledger files)
and I would like to write some automated tests to check the
functionality of the minor mode.  I found ERT, but it seems that it does
not offer any facility to easily test fontification, indentation, or
buffer manipulation in general.

Is there any facility that would help in writing such tests?

You might want to look at ledger-mode tests. Its approach to
fontification testing is:

* Create a temporary buffer.
* Put test text into it.
* Harvest it back with fontification properties into a data structure.
* Test that this structure is equal to the golden output.


Thanks Yuri, this is very helpful.

Cheers,
Dan




Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Richard Stallman
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If assess is useful, can we arrange to get it included in Emacs?

--
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Phillip Lord-3
Richard Stallman <[hidden email]> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If assess is useful, can we arrange to get it included in Emacs?

Yes. Been meaning too for ages.

I was hoping to have it as both an ELPA package and available in core,
but there still isn't a good mechanism for that at the moment. I guess I
should just add it to core, so it is available in Emacs-27.

Phil

Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Stefan Monnier
> I was hoping to have it as both an ELPA package and available in core,
> but there still isn't a good mechanism for that at the moment.

Don't know about "good", but we can (and do) publish GNU ELPA packages
where the code is kept in core (i.e. in the master branch of emacs.git):

    % grep :core .../elpa/externals-list
    ;;  :core     = part of GNU Emacs repository.
    ;; :core URL must be a list of:
     ;; ("cc-mode"          :core ("lisp/progmodes/cc-align.el"
     ("cl-print"            :core "lisp/emacs-lisp/cl-print.el")
     ("flymake"             :core "lisp/progmodes/flymake.el")
     ("jsonrpc"             :core "lisp/jsonrpc.el")
     ("let-alist"           :core "lisp/emacs-lisp/let-alist.el")
     ("map"                 :core "lisp/emacs-lisp/map.el")
     ("ntlm"                :core "lisp/net/ntlm.el")
     ("python"              :core "lisp/progmodes/python.el")
     ("soap-client"         :core ("lisp/net/soap-client.el" "lisp/net/soap-inspect.el"))
     ;;("tramp"             :core
    %

It has some rough edges, which makes it currently impractical to do that
for Tramp (for example), but for simple enough cases it works OK.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Phillip Lord-3
Stefan Monnier <[hidden email]> writes:

>> I was hoping to have it as both an ELPA package and available in core,
>> but there still isn't a good mechanism for that at the moment.
>
> Don't know about "good", but we can (and do) publish GNU ELPA packages
> where the code is kept in core (i.e. in the master branch of emacs.git):
>
>     % grep :core .../elpa/externals-list
>     ;;  :core     = part of GNU Emacs repository.
>     ;; :core URL must be a list of:
>      ;; ("cc-mode"          :core ("lisp/progmodes/cc-align.el"
>      ("cl-print"            :core "lisp/emacs-lisp/cl-print.el")
>      ("flymake"             :core "lisp/progmodes/flymake.el")
>      ("jsonrpc"             :core "lisp/jsonrpc.el")
>      ("let-alist"           :core "lisp/emacs-lisp/let-alist.el")
>      ("map"                 :core "lisp/emacs-lisp/map.el")
>      ("ntlm"                :core "lisp/net/ntlm.el")
>      ("python"              :core "lisp/progmodes/python.el")
>      ("soap-client"         :core ("lisp/net/soap-client.el" "lisp/net/soap-inspect.el"))
>      ;;("tramp"             :core
>     %
>
> It has some rough edges, which makes it currently impractical to do that
> for Tramp (for example), but for simple enough cases it works OK.


Yeah, I saw that, and that would be a good way to go. I guess these
packages are developed in core though (i.e. it's their primary
repository)? For assess, it's nice to be able to develop them outside of
core, because that makes it easier, for example, to test against
multiple versions of Emacs (I am trying to support two major versions).

I've also got working code that takes an ELPA package and copies it into
the core build. It's quite simple to do actually, the documentation
needs improving and to plump in some configure options. It would raise
the possibility of having a "Emacs minimal" core distribution (i.e. with
no "elpa" packages). Interested?

Phil

Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Ted Zlatanov
On Thu, 24 Jan 2019 10:29:29 +0000 [hidden email] (Phillip Lord) wrote:

PL> I've also got working code that takes an ELPA package and copies it into
PL> the core build. It's quite simple to do actually, the documentation
PL> needs improving and to plump in some configure options. It would raise
PL> the possibility of having a "Emacs minimal" core distribution (i.e. with
PL> no "elpa" packages). Interested?

That would be a terrific addition to emba.gnu.org (currently it just
does the default build):

* build and test without ELPA
* build and test with ELPA (including tests from ELPA!)

Network connectivity is slightly tricky because it's through a proxy but
may not be needed to ELPA. Let me know if that sounds useful and I can
try to implement the CI side.

Thanks :)
Ted

Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Stefan Monnier
In reply to this post by Phillip Lord-3
> I've also got working code that takes an ELPA package and copies it into
> the core build.  It's quite simple to do actually, the documentation
> needs improving and to plump in some configure options.  It would raise
> the possibility of having a "Emacs minimal" core distribution (i.e. with
> no "elpa" packages).  Interested?

I think we need that, yes (e.g. we really should bundle Company with
Emacs while we want to keep it in elpa.git).  But it's for Eli to decide
if and how he wants such a thing.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Eli Zaretskii
> From: Stefan Monnier <[hidden email]>
> Date: Fri, 25 Jan 2019 19:46:20 -0500
>
> > I've also got working code that takes an ELPA package and copies it into
> > the core build.  It's quite simple to do actually, the documentation
> > needs improving and to plump in some configure options.  It would raise
> > the possibility of having a "Emacs minimal" core distribution (i.e. with
> > no "elpa" packages).  Interested?
>
> I think we need that, yes (e.g. we really should bundle Company with
> Emacs while we want to keep it in elpa.git).  But it's for Eli to decide
> if and how he wants such a thing.

My opinions on that are known: I think that we should have everything
useful in core, and ELPA should only host packages that are "not yet"
in core, or have some other issues that prevent their move to core.
Since this is at odds with opinions of many others, I very much doubt
that my decision along these lines will stand.

Reply | Threaded
Open this post in threaded view
|

Re: Testing fontification, indentation, and buffer manipulation

Phillip Lord-3
In reply to this post by Stefan Monnier
Stefan Monnier <[hidden email]> writes:

>> I've also got working code that takes an ELPA package and copies it into
>> the core build.  It's quite simple to do actually, the documentation
>> needs improving and to plump in some configure options.  It would raise
>> the possibility of having a "Emacs minimal" core distribution (i.e. with
>> no "elpa" packages).  Interested?
>
> I think we need that, yes (e.g. we really should bundle Company with
> Emacs while we want to keep it in elpa.git).  But it's for Eli to decide
> if and how he wants such a thing.

I've pushed this to:

feature/core-elpa-by-copy

Currently, it's in as a feature

./configure --enable-elpa
./configure --enable-elpa=where-to-find-or-clone-elpa

Main features:

 - Works with externals or non-external packages.
 - Versions are specified by git SHA, so a version build will be
   repeatable.
 - Both code and tests are supported
 - It's customizable by package authors on ELPA.
 - Doesn't use package.el (I think this is an anti-feature, but tried
   that before and no one liked it)
 - It requires network access only if ELPA is not available locally, or if
   the Makefile which describes versions is updated
 - Works with make -j in which case it does a git clone which the C is building

None Features:
 - Currently, only works with a single git repo (i.e. ELPA) which is
   hard coded.
 - Doesn't support documentation or info

To be done:
 - Give it prettier output
 - Clean ups.


Phil

Reply | Threaded
Open this post in threaded view
|

Core ELPA was: Testing fontification, indentation, and buffer manipulation

Phillip Lord-3
[hidden email] (Phillip Lord) writes:

> Stefan Monnier <[hidden email]> writes:
>
>>> I've also got working code that takes an ELPA package and copies it into
>>> the core build.  It's quite simple to do actually, the documentation
>>> needs improving and to plump in some configure options.  It would raise
>>> the possibility of having a "Emacs minimal" core distribution (i.e. with
>>> no "elpa" packages).  Interested?
>>
>> I think we need that, yes (e.g. we really should bundle Company with
>> Emacs while we want to keep it in elpa.git).  But it's for Eli to decide
>> if and how he wants such a thing.
>
> I've pushed this to:
>
> feature/core-elpa-by-copy
>
> Currently, it's in as a feature
>
> ./configure --enable-elpa
> ./configure --enable-elpa=where-to-find-or-clone-elpa
>
> Main features:
>
>  - Works with externals or non-external packages.
>  - Versions are specified by git SHA, so a version build will be
>    repeatable.
>  - Both code and tests are supported
>  - It's customizable by package authors on ELPA.
>  - Doesn't use package.el (I think this is an anti-feature, but tried
>    that before and no one liked it)
>  - It requires network access only if ELPA is not available locally, or if
>    the Makefile which describes versions is updated
>  - Works with make -j in which case it does a git clone which the C is building
>
> None Features:
>  - Currently, only works with a single git repo (i.e. ELPA) which is
>    hard coded.
>  - Doesn't support documentation or info
>
> To be done:
>  - Give it prettier output
>  - Clean ups.


Wondering whether anyone had time to give me feedback on this. It
provides a mechanism to have ELPA packages bundle with core Emacs.

Phil


Reply | Threaded
Open this post in threaded view
|

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

Stefan Monnier
> Wondering whether anyone had time to give me feedback on this. It
> provides a mechanism to have ELPA packages bundle with core Emacs.

Sorry for taking so long.  I just took a look at it, and it looks fairly
simple and clean.  But I think the main question is *if* we really want
to use this to build the official tarballs.  I think we should, but IIRC
Eli wasn't quite as enthusiastic about it.

>  - Doesn't use package.el (I think this is an anti-feature, but tried
>    that before and no one liked it)

That doesn't ring a bell.  Could you refresh our memory why no one
liked it?

> --- /dev/null
> +++ b/notes.org
> @@ -0,0 +1,84 @@
> +
> +
> +* Working on
> +
> +Have package checkout working for external
> +Next get it working for non external so we can template

Don't know what you mean by "can template"

> +* To Fix
> +Think we are running git fetch --all on every package.

Don't know what you're referring to here either.

> +1) Scripting?
> +Guess we have to use sh

Of course, we could also use Elisp ;-)

> +2) How to get specific directory of a repository
> +
> +
> +
> +git archive --remote=[hidden email]:/srv/git/emacs/elpa.git master packages/hydra | tar xv --strip-components=1
> +
> +Currently, this requires a member login -- it doesn't work with http
> +or the current git server.
> +
> +Would also be possible to configure this from local.

I'm not familiar with `git archive` but does this require network access?

> +3) How to work out when a directory is out of date wrt to the make
> +   file

Don't know what you mean here.

> +4) Heuristics for working out which files to copy to right place
> +
> +*-test.el
> +*-tests.el
> +/test/*.el
> +
> +to test
> +
> +All other *el to lisp
> +
> +Nothing else

Rather than heuristics, I'd go for a *rules* (which needs to be obeyed
before the package can be bundled).

But I'm not sure we should install those packages with a different
layout than for a "normal" ELPA install.
What's the advantage of using a different layout?

> +6) How to enable ELPA packages to customize the process
> +Add the option to run an core-deploy.sh file placed into the root of a
> +package.

Don't know what you mean here.

> +8) Support other repos
> +Org-mode effectively does this already

Don't know what you mean here.

> --- /dev/null
> +++ b/elpa/bin/extract-package.sh
> @@ -0,0 +1,55 @@
> +#!/bin/bash
> +
> +function grab_external {
> +    rm -rf packages/$PACKAGE*
> +    mkdir --parents $PACKAGE_LOC
> +    cwd=`pwd`
> +    cd $GIT_LOC
> +    git archive $SHA \
> +        | tar xv -C $cwd/$PACKAGE_LOC
> +    cd $cwd
> +    cp --no-clobber bin/package-makefile.mk $PACKAGE_LOC

Rather than `cd $cwd`, I generally prefer to use (...) to save&restore
the current directory:

    cwd=`pwd`
    (cd $GIT_LOC
     git archive $SHA \
         | tar xv -C $cwd/$PACKAGE_LOC)
    cp --no-clobber bin/package-makefile.mk $PACKAGE_LOC

If we keep the layout unchanged, we could use `git worktree add`, which
has the advantage that when running Emacs in-place, jumping to the
source in `C-h f` and friends gives you access to the Git metainfo.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

Phillip Lord-3
Stefan Monnier <[hidden email]> writes:

>> Wondering whether anyone had time to give me feedback on this. It
>> provides a mechanism to have ELPA packages bundle with core Emacs.
>
> Sorry for taking so long.  I just took a look at it, and it looks fairly
> simple and clean.  But I think the main question is *if* we really want
> to use this to build the official tarballs.  I think we should, but IIRC
> Eli wasn't quite as enthusiastic about it.

Indeed. It also means that git will (sometimes) be needed to build;
although this will only be true after git pulling the main git repo, so
perhaps this is not such an issue.

I would also plan to build two tar balls -- emacs-and-selected-elpa.tgz
and emacs.tgz. Over time, I think it would make sense to change these to
emacs.tgz and emacs-minimal.tgz (i.e. with ELPA becomes the default).


>>  - Doesn't use package.el (I think this is an anti-feature, but tried
>>    that before and no one liked it)
>
> That doesn't ring a bell.  Could you refresh our memory why no one
> liked it?

A variety of reasons. It's less simple. It requires changes to
package.el (for example, what should emacs -Q do with ELPA packages. In
my original version they would be disabled, in this version they are
still on). package.el has to be launched every start up for every
emacs. And Eli in particular didn't like the increase in the number of
directories in the default load-path. This implementation does the same
thing, but it doesn't have to -- we could copy all the lisp files into
the same directory.



>> --- /dev/null
>> +++ b/notes.org
>> @@ -0,0 +1,84 @@
>> +
>> +
>> +* Working on
>> +
>> +Have package checkout working for external
>> +Next get it working for non external so we can template
>
> Don't know what you mean by "can template"

Sorry the notes are a bit out of date (bad habit of mine, and I wasn't
expecting anyone to read this file, cause it will go before merge).

This is working now (neither queue nor darkroom are external).


>> +* To Fix
>> +Think we are running git fetch --all on every package.
>
> Don't know what you're referring to here either.


Also fixed. Getting the dependencies was a bit tricky.


>> +1) Scripting?
>> +Guess we have to use sh
>
> Of course, we could also use Elisp ;-)

If I remember correctly, all of this happens before the full Emacs is
dumped. It seemed cleaner to do this to me.


>> +2) How to get specific directory of a repository
>> +
>> +
>> +
>> +git archive
>> --remote=[hidden email]:/srv/git/emacs/elpa.git
>> master packages/hydra | tar xv --strip-components=1
>> +
>> +Currently, this requires a member login -- it doesn't work with http
>> +or the current git server.
>> +
>> +Would also be possible to configure this from local.
>
> I'm not familiar with `git archive` but does this require network
> access?

git archive gives a bare, not with a .git, directory.

I had to drop that command anyway. Git doesn't allow checkout of an
arbitrary SHA from a remote repo (because it would allow checking out
commits that had been deleted otherwise, I believe). So instead, now,
the makefile checkouts out the whole of ELPA, and then uses git archive
to get a specific SHA from local.

Obviously, this is a fairly network intensive way of getting just a few
files; but it's a fixed sized problem.


>> +3) How to work out when a directory is out of date wrt to the make
>> +   file
>
> Don't know what you mean here.


That's fixed also. Packages should updated from git (and git fetch run)
if Makefile is updated. So, this means a git fetch after everytime Emacs
is ./configure is run.

>
>> +4) Heuristics for working out which files to copy to right place
>> +
>> +*-test.el
>> +*-tests.el
>> +/test/*.el
>> +
>> +to test
>> +
>> +All other *el to lisp
>> +
>> +Nothing else
>
> Rather than heuristics, I'd go for a *rules* (which needs to be obeyed
> before the package can be bundled).

ELPA already uses heuristics (see admin/ert-support.el). Rules would
make this simpler, of course, at the cost of, well, another rule.


> But I'm not sure we should install those packages with a different
> layout than for a "normal" ELPA install.
> What's the advantage of using a different layout?

I haven't checked to see who uses them or why, so I don't know.

>
>> +6) How to enable ELPA packages to customize the process
>> +Add the option to run an core-deploy.sh file placed into the root of a
>> +package.
>
> Don't know what you mean here.


That bits out of date, and supported another way.

By default, EMACS/elpa/bin/package-makefile.mk is copied into a package
after it is pulled out of git and it is this that copies files from the
elpa directory into the core Emacs layout. If a package wants to
override this behaviour, it could include it's own package-makefile.mk.


>> +8) Support other repos
>> +Org-mode effectively does this already
>
> Don't know what you mean here.

Currently, this will only take files from the ELPA repo, but there is
not reason that this needs to be so. Files could come another repo (such
as org-mode's own). My guess is that there is no particular reason to
support this at the moment.


>> --- /dev/null
>> +++ b/elpa/bin/extract-package.sh
>> @@ -0,0 +1,55 @@
>> +#!/bin/bash
>> +
>> +function grab_external {
>> +    rm -rf packages/$PACKAGE*
>> +    mkdir --parents $PACKAGE_LOC
>> +    cwd=`pwd`
>> +    cd $GIT_LOC
>> +    git archive $SHA \
>> +        | tar xv -C $cwd/$PACKAGE_LOC
>> +    cd $cwd
>> +    cp --no-clobber bin/package-makefile.mk $PACKAGE_LOC
>
> Rather than `cd $cwd`, I generally prefer to use (...) to save&restore
> the current directory:
>
>     cwd=`pwd`
>     (cd $GIT_LOC
>      git archive $SHA \
>          | tar xv -C $cwd/$PACKAGE_LOC)
>     cp --no-clobber bin/package-makefile.mk $PACKAGE_LOC

Oh, yes, hadn't thought of that.


> If we keep the layout unchanged, we could use `git worktree add`, which
> has the advantage that when running Emacs in-place, jumping to the
> source in `C-h f` and friends gives you access to the Git metainfo.

Not sure I have understood -- you mean git subtree add perhaps? This
wouldn't work anyway, because C-h f will jump to source in
EMACS/lisp/elpa which is not created by git archive.

Phil

Reply | Threaded
Open this post in threaded view
|

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

Stefan Monnier
>>> +1) Scripting?
>>> +Guess we have to use sh
>> Of course, we could also use Elisp ;-)
> If I remember correctly, all of this happens before the full Emacs is
> dumped.  It seemed cleaner to do this to me.

I'm fine with using `sh`, but I think that it'd be perfectly OK to delay
this elpa-inclusion to after `src/emacs` is dumped.

> That's fixed also. Packages should updated from git (and git fetch run)
> if Makefile is updated. So, this means a git fetch after everytime Emacs
> is ./configure is run.

I think a plain `make` shouldn't perform a `git fetch` every time the
Makefile changes.  It's OK to require a `git fetch` in that case, but
if/when the `git fetch` happens should be under the control of the user
(i.e. should require some explicit step like `make update-elpa`).

>> But I'm not sure we should install those packages with a different
>> layout than for a "normal" ELPA install.
>> What's the advantage of using a different layout?
> I haven't checked to see who uses them or why, so I don't know.

I think there's a misunderstanding: I was not talking about the layout
used internally by the packages, but the layout used by your code (where
you put some stuff in lisp/ and other in test/ so the layout of the
package's files is modified compared to what happens in a normal ELPA
install).

> Currently, this will only take files from the ELPA repo, but there is
> not reason that this needs to be so. Files could come another repo (such
> as org-mode's own). My guess is that there is no particular reason to
> support this at the moment.

I see no need to support fetching from other Git repositories, indeed,
except in order to use a local mirror, of course.

> This wouldn't work anyway, because C-h f will jump to source in
> EMACS/lisp/elpa which is not created by git archive.

What I meant is that it'd be much better if we could make sure that
EMACS/lisp/elpa has the relevant .git metadata.


        Stefan

Reply | Threaded
Open this post in threaded view
|

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

Phillip Lord-3
Stefan Monnier <[hidden email]> writes:

>>>> +1) Scripting?
>>>> +Guess we have to use sh
>>> Of course, we could also use Elisp ;-)
>> If I remember correctly, all of this happens before the full Emacs is
>> dumped.  It seemed cleaner to do this to me.
>
> I'm fine with using `sh`, but I think that it'd be perfectly OK to delay
> this elpa-inclusion to after `src/emacs` is dumped.

The `sh` is written now anyway and it's fairly simple, so I am less
worried about it. It does make the dependencies easier, which Emacs
multi-directory structure doesn't make easier. Especially, since the git
clone (or update as necessary) currently happens during the C build. In
practice, this means the network time has a small impact on the overall build.


>> That's fixed also. Packages should updated from git (and git fetch run)
>> if Makefile is updated. So, this means a git fetch after everytime Emacs
>> is ./configure is run.
>
> I think a plain `make` shouldn't perform a `git fetch` every time the
> Makefile changes.  It's OK to require a `git fetch` in that case, but
> if/when the `git fetch` happens should be under the control of the user
> (i.e. should require some explicit step like `make update-elpa`).

The problem is that if the Makefile updates, then it is possible that
it contains an sha-1 that doesn't exist in the ELPA clone, because the
makefile is new. If this happens the git archive command will break. I'd
rather have the build just work. Apart from convienience, if it does
not, emacs-devel will get a lot of "emacs build is broken" errors after
an update.

I could move the dependency to Makefile.in I guess? Then a simple
./configure wouldn't change things, but any textual change in
Makefile.in would. Or, I could check the repo for the SHA-1 first and if
this doesn't exist, the run git fetch.




>>> But I'm not sure we should install those packages with a different
>>> layout than for a "normal" ELPA install.
>>> What's the advantage of using a different layout?
>> I haven't checked to see who uses them or why, so I don't know.
>
> I think there's a misunderstanding: I was not talking about the layout
> used internally by the packages, but the layout used by your code (where
> you put some stuff in lisp/ and other in test/ so the layout of the
> package's files is modified compared to what happens in a normal ELPA
> install).


Oh, sorry. Think we are good, then. At the moment, there is a single
layout for any ELPA package (unless it provides it's own
package-makefile.mk).

>> Currently, this will only take files from the ELPA repo, but there is
>> not reason that this needs to be so. Files could come another repo (such
>> as org-mode's own). My guess is that there is no particular reason to
>> support this at the moment.
>
> I see no need to support fetching from other Git repositories, indeed,
> except in order to use a local mirror, of course.

Yes!



>> This wouldn't work anyway, because C-h f will jump to source in
>> EMACS/lisp/elpa which is not created by git archive.
>
> What I meant is that it'd be much better if we could make sure that
> EMACS/lisp/elpa has the relevant .git metadata.


So that you could edit EMACS/lisp/elpa files in their installed location
and update them? Or something simpler. I think that would be a big
challenge.

Simpler would be to add a file called ".this-is-not-a-source-file" and
have emacs-lisp-mode detect this and point at the real
source. My default my Makefile makes a bare clone of ELPA so there are
no source files to edit there but

./configure --enable-elpa=my-real-no-bare-elpa-clone

should also work. So there might well be real source.

Phil

Reply | Threaded
Open this post in threaded view
|

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

Stefan Monnier
> I could move the dependency to Makefile.in I guess? Then a simple
> ./configure wouldn't change things, but any textual change in
> Makefile.in would. Or, I could check the repo for the SHA-1 first and if
> this doesn't exist, the run git fetch.

My opinion is that the problem in in having the SHAs in Makefile.in.
I think Makefile.in should refer to branches/tags rather than to SHAs.


        Stefan


PS: Also, every time Makefile.in changes, `make` re-runs config.status
which then causes all the .o files to be recompiled as well, so I'd
rather we don't change it all too often.


Reply | Threaded
Open this post in threaded view
|

Re: Core ELPA was: Testing fontification, indentation, and buffer manipulation

Richard Stallman
In reply to this post by Stefan Monnier
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think a plain `make` shouldn't perform a `git fetch` every time the
  > Makefile changes.  It's OK to require a `git fetch` in that case, but
  > if/when the `git fetch` happens should be under the control of the user
  > (i.e. should require some explicit step like `make update-elpa`).

I don't think it is right for 'make' to affect the source files people edit,
except in limited automatic ways.

There are sometimes reasons to continue building the same checkout --
perhaps making some local edits -- and excluding all other changes
that could confuse matters.
--
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



123