Friendly discussion about (package-initialize)

classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Friendly discussion about (package-initialize)

Radon Rosborough
Hi all,

I'd like to have a friendly discussion about (package-initialize).
Specifically, about this comment that I'm sure we've all seen from
time to time:

    ;; Added by Package.el.  This must come before configurations of
    ;; installed packages.  Don't delete this line.  If you don't want it,
    ;; just comment it out by adding a semicolon to the start of the line.
    ;; You may delete these explanatory comments.
    (package-initialize)

In this email, I'm going to explain why I think the current behavior
of package.el is flawed, and propose some alternative behaviors that I
believe would be better. If the community thinks that these
alternatives are reasonable, I would be interested in contributing
relevant patches. I know that this argument has the potential to be
contentious, so I'm going to try to be as objective as possible, and
stick strictly to the technical points.

Since this is a rather long email, I've arranged it into sections.
Feel free to skip or skim the parts that you already know. I've tried
to be quite verbose so as to create a comprehensive and authoritative
summary of the issue for people who may not be as familiar with Emacs
packaging.

################################################################################
# WHAT DOES PACKAGE.EL DO?

Since version 24.1 [1], Emacs has shipped with a built-in package
manager called package.el [2]. Like other package managers such as
Quelpa [3], Cask [4], el-get [5], Borg [6], and straight.el [7], the
primary function of package.el is to acquire Elisp code, preprocess it
(e.g. byte-compilation, autoload generation), and make it available to
Emacs.

The acquiry and preprocessing of Elisp code is not relevant to this
discussion. What is important is how the code is made available to
Emacs. The general process (called "activating" a package) is
two-fold:

* add the package's files to the load-path and Info-directory-list
* evaluate the package's autoloads, so that running an autoloaded
  function loads the appropriate files from the package

Importantly, these steps must be run every time Emacs is initialized.
Installing Elisp code on the filesystem does nothing, unless it is
somehow added to one of Emacs' default load-path entries. Before a
package is activated, attempts to `require' its features will fail,
and its autoloads will not be available.

Different package managers have different opinions on how and when
packages ought to be activated. For example, in straight.el [7],
packages are activated one at a time as they are referenced in the
user's init-file. However, the most common approach is for all
packages to be activated at the same time. This is what package.el
does; it provides a `package-initialize' function which fulfills this
purpose.

Therefore, absent any special treatment, one might expect a user's
init-file to go something like this:

    ... code to configure package.el ...
    (package-initialize '(... some list of packages ...))
    ... code to configure the packages that were initialized ...

But users do not typically think in this way. They expect that
installing a package is a permanent operation, and that no further
action is required in order for that package to continue to be
available. For this reason, package.el keeps track of which packages
have been installed already, and it does this outside the init-file
(in the state of the ~/.emacs.d/elpa directory). When
`package-initialize' is called, all the installed packages are
activated. You can request package.el to activate only a subset of the
installed packages, but this is not the usual mode of operation. Thus,
the user's init-file actually looks like this:

    ... code to configure package.el ...
    (package-initialize)
    ... code to configure the packages that were initialized ...

Notice that the call to `package-initialize' must come *after* any
code that configures package.el, but *before* any code that configures
the packages that were initialized. This is important! If you try to
configure package.el after `package-initialize', your configurations
will not affect how package activation takes place. And if you try to
configure packages before `package-initialize', then variables,
functions, and so on are not going to be properly defined. (This might
not be a problem, depending on how you handle your customizations. But
it certainly does make things more complicated.)

################################################################################
#### WHAT IS THE PROBLEM?

The problem is when users forget to initialize the package management
system before configuring their packages. In other words, their
init-file just looks like this:

    ... code to configure packages ...

with no mention of `package-initialize' anywhere. Obviously this will
not work, because the packages are never activated. The "correct"
solution is to just call `package-initialize' in the init-file. But it
is hard to teach users to do this, and the resulting errors have been
a common problem for new Emacs users [8].

Thus, there was a discussion [8] on emacs-devel about how to arrange
for things to work "correctly" in the absence of a user who knows what
they are doing. The eventual outcome of that discussion is the subject
of the next section.

################################################################################
#### WHAT IS THE CURRENT SOLUTION TO THE PROBLEM?

The initial attempt was to have Emacs call `package-initialize'
automatically during initialization. However, this is not so easy as
it sounds. If Emacs calls `package-initialize' before loading the
init-file, then users cannot customize package.el in their init-file
anymore. And if Emacs calls `package-initialize' after loading the
init-file, then users cannot customize their installed packages in
their init-file anymore, unless they cause their customizations to be
loaded after the end of initialization using `after-init-hook'.

The only reasonable way for things to be set up is for
`package-initialize' to be called in the user's init-file, but before
package customizations are done. Thus, a patch [9] was suggested that
causes Emacs to write a call to `package-initialize' into the user's
init-file at startup, if it was not there to begin with. This was
shortly merged and released in Emacs 25.1. Note that the behavior is
inhibited in 'emacs -Q' mode.

################################################################################
#### JUSTIFICATIONS AND COUNTERARGUMENTS

> So what are you saying is wrong with the current behavior?

It modifies the user's init-file without asking.

> What's wrong with modifying the user's init-file? After all, Custom
> has been doing that for decades.

The difference is that Custom only modifies your init-file when you
ask it to save your customizations. In other words, it only modifies
your init-file when you ask it to. On the other hand, package.el does
it at Emacs startup without confirmation!

> OK, so it's different than Custom. What's the problem, though?

There are several issues:

* Many users want to write their init-files by hand. If they want to
  initialize the package management system, they want to do it
  themselves, and they want to put in the code by hand so that they
  know it will run in the right place.
* Many users have a modular Emacs configuration in which init.el loads
  their other customization files, one of which is usually deals with
  the package management system (including calling
  `package-initialize'). The result is that package.el sticks a
  superfluous, duplicate `package-initialize' into the root init-file.
* It's confusing. New users have no idea why Emacs is modifying their
  init-file, and a proper explanation is not given.
* It's extremely error-prone. Modifying the init-file is an
  irreversible operation, and it means that any bugs in this part of
  package.el are immediately 100x more frustrating.
* Not everyone uses package.el. It's annoying to have to explicitly
  put in configuration to disable the existing package management
  system, if you want to use an alternative package manager.

> Slow down there! Let's take this one point at a time. You said that
> some users want to write their init-files by hand. But if they do
> that, then can't they just put a call to `package-initialize' in
> their init-files, and that will prevent the issue?

No, because (1) they might be using a modular Emacs configuration that
calls `package-initialize' somewhere other than init.el, and (2) they
might be using a package manager other than package.el, and therefore
have no need of calling `package-initialize' at all.

> Well, what's wrong with them just putting a commented-out call to
> `package-initialize' in their init-file, like the inserted comments
> suggest?

This is ugly and looks ridiculous. Is requiring a magic comment in the
init-file to prevent unnecessary code from being inserted on Emacs
startup really the best possible user experience?

> Then what would be a better user experience?

Let's wait until the ALTERNATIVES section for that :)

> Wait, you said that it was a problem if `package-initialize' was put
> somewhere other than in the init-file. But if it's called during
> startup, then package.el doesn't run the logic to insert the a call
> into the init-file. So what's the issue?

The problem is that if there is an error while loading the init-file,
then the call to `package-initialize' still gets inserted, if init
hadn't gotten to the point where the actual call to
`package-initialize' was supposed to happen. This is actually
terrible, since it immediately makes debugging much more complicated:
your init-file is being changed without your knowing it, while you're
trying to debug initialization!

> OK, but that only happens when there's an error during init. We
> could make it so that the `package-initialize' happens only after a
> successful init.

That doesn't solve the problem, because there are all kinds of
circumstances where you have a "successful" init even when the user's
init-file was not loaded. For example, many modular Emacs
configurations will sustain errors in one or more modules using
`condition-case', and report the errors as warnings for an improved
debugging experience. We're back to the beginning.

Besides, often one wants to test something in a plain Emacs but not in
'emacs -Q'. In this case, the call to `package-initialize' will still
get inserted.

> How often does this really happen?

At least 20 times in my use of Emacs, and I get progressively more
annoyed every time. The existence of threads like [10] [11] indicates
that there are other people who don't like this kind of behavior.

Regardless of how we set up this system, it will never be possible to
cover all the edge cases. Once in a while, there will be some
sufficiently unexpected use case where an errant `package-initialize'
gets inserted and screws something up. Since there are better
solutions (see ALTERNATIVES), I think it's best to sidestep the whole
problem by not taking this approach in the first place.

> You said the system was confusing. Why?

The comments that come with the inserted `package-initialize' tell the
user "don't delete this line" without explaining the consequences if
you do. The user inevitably will try deleting it, since they didn't
ask for it to be added in the first place. But it will come back next
time, prompting annoyance and consternation.

The comments also advise "this must come before configurations of
installed packages", without explaining why or anything about how
package activation works. Also, the whole idea of "if you don't want
this, comment it out, but don't remove it" is bizarre; I've never seen
anything like it in any other program's configuration system.

> Can't that be fixed by clearer comments?

Yes, but the fact remains that a program inserting text into its own
configuration file (except in the case where the program has a
semantic understanding of the whole file, like for YAML) is highly
nonstandard. I don't know of any other program that does something
like this. As I said before, there are ALTERNATIVES that are more
standard, and these will not violate the principle of least surprise
as badly.

> Why are you so concerned about bugs in package.el? Is that really
> likely?

I trust that the folks here on emacs-devel will do a good job in
preventing bugs from creeping in. But nevertheless, I think putting in
this kind of logic is simply a bad idea from a software engineering
perspective. There will always be bugs, and we want to minimize the
impact of the ones that inevitably do pop up. Having code that
automatically modifies the init-file at startup is setting up the
potential for a bug with a huge impact that will be extremely annoying
and dangerous to work around. I think this is especially important to
keep in mind since Emacs releases are extremely infrequent, and so
we're going to have to deal with bugs in any release version for a
long, long time.

> What's this about alternative package managers? Shouldn't Emacs
> include special support for the built-in one? After all, that's why
> it's built in.

Sure, I think it's reasonable for Emacs to provide special support for
packages which are built in. But there's such a thing as going too
far. And I personally think that you've gone too far in providing
special support when that support actively makes it *more* difficult
to swap out an alternative implementation.

If you use an alternative package manager (and many people do), then
you really don't want package.el to be automatically activating things
when it's not asked to. You'll end up with duplicate load-path
entries, conflicting versions, needlessly slow init times, and more:
fun times (I've been there).

And you *really* don't want package.el to be modifying your init-file
that configures your alternative package manager so that it also
activates package.el! Having such code inserted when it's actively
harmful is a really frustrating experience.

################################################################################
#### ALTERNATIVES

There is lots we can do to improve the situation. Some changes will be
harmless and backwards compatible, while others would have to be
rolled out carefully if we decide to follow them.

* Don't have package.el insert `package-initialize' into the init-file
  under any circumstances. Instead, when Emacs is started up for the
  first time (when not in 'emacs -Q' mode), if no file exists at
  ~/.emacs.d/init.el, create that file with a basic template.

  This template would have a call to `package-initialize', and also
  some explanatory comments saying where you should put configuration
  of package.el versus configuration of packages, and why.

  This is my favorite option. It is a well-established (and,
  importantly, *unsurprising*) tradition for programs to come with
  skeleton configuration files that the user can customize as they
  desire. Besides, this would allow us to make a better experience for
  new users (remember all the complaints about "bad defaults"?)
  without making actual changes to the defaults and breaking backward
  compatibility.

* Don't have package.el activate packages at startup. I would actually
  prefer this, but I understand that it's not likely to happen. There
  are two advantages: (1) users of alternative package managers don't
  have to explicitly disable package.el all the time (when testing
  without their configuration file, for example), and (2) the behavior
  is much less "magic".

  This would break backward compatibility, but mostly of expectations:
  only people who put their entire package configuration code into
  `after-init-hook' would have their configurations broken (and the
  fix would of course be trivial). If we implemented the skeleton init
  file, then the experience for new users would not change.

* Improve the explanatory value of the comments that are inserted by
  package.el. I really, really think that the whole behavior of
  automatically inserting code into the init-file should be killed
  (with fire), but if the community really disagrees with me here, at
  least the comments should make an attempt at explaining what's going
  on and what the potential caveats are. This would be a trivial
  change with no downsides.

* Make it trivial to disable package.el. Right now, the only foolproof
  way to prevent the init-file from being modified by package.el is to
  place multiple advices on internal functions, and to do this as
  early as possible during init just in case there is an error, and to
  also be really careful any time you load Emacs without your
  init-file, and to double-check for the inevitable times when you
  can't prevent package.el from doing stuff.

################################################################################
#### CONCLUSION AND NEXT STEPS

I've argued strongly for removing the auto-insertion of
`package-initialize' code into the init-file by package.el, in favor
of providing a skeleton init-file. Although I've addressed a number of
counterarguments, I'm sure there are other factors I haven't
considered. Let me know what you think. Is this reasonable? If not,
why not? If so, how can I help move it forward?

Best regards,
Radon Rosborough

[1]: https://lists.gnu.org/archive/html/info-gnu-emacs/2012-06/msg00000.html
[2]: https://www.gnu.org/software/emacs/manual/html_node/emacs/Packages.html
[3]: https://github.com/quelpa/quelpa
[4]: https://github.com/cask/cask
[5]: https://github.com/dimitri/el-get
[6]: https://github.com/emacscollective/borg
[7]: https://github.com/raxod502/straight.el
[8]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html
[9]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01055.html
[10]: https://www.reddit.com/r/emacs/comments/56fvgd/is_there_a_way_to_stop_emacs_from_adding_the/?ref=share&ref_source=link
[11]: https://www.reddit.com/r/emacs/comments/4x655n/packageselectedpackages_always_appear_after/
[12]: https://github.com/raxod502/straight.el/issues/73

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Stefan Monnier
> Sure, I think it's reasonable for Emacs to provide special support for
> packages which are built in. But there's such a thing as going too
> far.  And I personally think that you've gone too far in providing
> special support when that support actively makes it *more* difficult
> to swap out an alternative implementation.

You wrote lots and lots of lines of text just to complain about the
addition of a single "(package-initialize)" in the user's ~/.emacs,
which should do absolutely nothing in the case where the user doesn't
use package.el (i.e. doesn't have anything inside ~/.emacs.d/elpa).

So, it's not that big of a deal, really.

We could arrange for "(package-initialize)" to only be added if there is
at least one package inside ~/.emacs.d/elpa.

More to the point, I think that's already the case.  So users of other
package managers should simply never bump into this text (unless they
also user package.el, of course).  If they do, they should report it as
a bug.

> * Make it trivial to disable package.el. Right now, the only foolproof
>   way to prevent the init-file from being modified by package.el is to
>   place multiple advices on internal functions, and to do this as

Multiple?  Doesn't

    (advice-add 'package--ensure-init-file :override #'ignore)

do the trick?

I suspect that

    (setq package-enable-at-startup nil)

might also do the trick.

This said, the main motivation for calling package--ensure-init-file
from package-initialize was to fix existing user's config where they had
packages installed yet their .emacs didn't call package-initialize, so
they had trouble configuring their packages.  One might argue that this
situation is now mostly fixed and we could change tactic: only call
package--ensure-init-file when the user installs a package.

Another thing is that rather then look for "(package-initialize)" in
~/.emacs we could keep track of whether package-initialize was called
during initialization.  This will avoid the problem when the user placed
his call in another file.


        Stefan


PS: BTW, I'd be very interested to work with maintainers of other
package managers to see how we could make them better interoperate
(e.g. make it possible to install with one tool, but activate&config
with another).


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Radon Rosborough
> You wrote lots and lots of lines of text just to complain about the
> addition of a single "(package-initialize)" in the user's ~/.emacs,
> which should do absolutely nothing in the case where the user
> doesn't use package.el (i.e. doesn't have anything inside
> ~/.emacs.d/elpa).

The problem is that even if I don't use package.el, there may be some
stuff left in ~/.emacs.d/elpa from previous times. For example, let's
say that I am trying to write a comparison between my package manager
and package.el, so even though I don't usually use package.el, I use
it temporarily.

Then, after I'm done testing, I remove all traces of package.el from
my init-file. But if I don't remember to also remove the data
directory ~/.emacs.d/elpa, then (package-initialize) is *not* a no-op:
it also activates duplicate copies of a bunch of old packages.

> So, it's not that big of a deal, really.

Only if you're OK with leaving a superfluous line in your init-file
for no reason other than that package.el wants it there. I know for a
fact that I'm not the only person who is annoyed by this kind of
behavior: see [1] for example.

Also, if it's not a big deal to have packages automatically insert
code into the user's init-file, then why don't more packages do this?
If every package did this, it would be a disaster. Somehow every
package other than package.el has managed without this mechanism; I
think package.el can do the same.

> We could arrange for "(package-initialize)" to only be added if
> there is at least one package inside ~/.emacs.d/elpa.

That's a good idea but it won't fix the problem, in my opinion. My
expectation of editor plugins is that if I remove the code configuring
them, then they should no longer affect me. This is not the case with
package.el; you have to specifically jump through some extra hoops to
prevent its side-effects from resurrecting themselves in future Emacs
sessions (e.g. deleting ~/.emacs.d/elpa).

> So users of other package managers should simply never bump into
> this text (unless they also user package.el, of course). If they do,
> they should report it as a bug.

Users of other package managers will still encounter this text unless
they delete ~/.emacs.d/elpa. In light of this, I think we have a
documentation bug, since the manual doesn't display a clear notice
that if you want to use an alternative package manager, you !!must!!
delete ~/.emacs.d/elpa first. More to the point, I don't think such a
step should be necessary: what if I wanted to switch between package
managers? That shouldn't be hard; there aren't these kinds of
persistent side effects with any of the other Emacs package managers I
tried, only for package.el.

> Multiple?  Doesn't
>
>     (advice-add 'package--ensure-init-file :override #'ignore)
>
> do the trick?

The second advice I was referring to was preventing
`package--save-selected-packages' from modifying the init-file.
However, the question of `package-selected-packages` is an entirely
separate issue and I shouldn't have snuck it in here. Sorry about
that; pretend I said just one advice.

> I suspect that
>
>     (setq package-enable-at-startup nil)
>
> might also do the trick.

I don't consider `package-enable-at-startup' to be an acceptable
solution, since I might want to use a function from package.el just to
try it out. In that case, my init-file will still get modified without
my permission. The name of the game here is "persistent side effects".
In my opinion, a package manager shouldn't have any. Again, for all
the other Emacs package managers I've seen (including the one I
wrote), you can perform any operations you'd like with them, and after
restarting Emacs, you'll be back where you started provided that you
didn't put anything in your init-file. This also happens to be the
case with every Emacs *package* I've seen. The only exception is
package.el, and I think we should consider why (if?) package.el needs
to be an exception.

> We could change tactic: only call package--ensure-init-file when the
> user installs a package.

I don't think I would enjoy this either. Would you consider it an
acceptable solution to pop up a window or display a message telling
the user that they should put `package-initialize' in their init-file,
provided that we didn't have it get called during init?

> Another thing is that rather then look for "(package-initialize)" in
> ~/.emacs we could keep track of whether package-initialize was
> called during initialization. This will avoid the problem when the
> user placed his call in another file.

Actually, Emacs already does this. And it doesn't solve the problem;
quoting from above:

    > Wait, you said that it was a problem if `package-initialize' was
    > put somewhere other than in the init-file. But if it's called
    > during startup, then package.el doesn't run the logic to insert
    > the a call into the init-file. So what's the issue?

    The problem is that if there is an error while loading the
    init-file, then the call to `package-initialize' still gets
    inserted, if init hadn't gotten to the point where the actual call
    to `package-initialize' was supposed to happen. This is actually
    terrible, since it immediately makes debugging much more
    complicated: your init-file is being changed without your knowing
    it, while you're trying to debug initialization!

    > OK, but that only happens when there's an error during init. We
    > could make it so that the `package-initialize' happens only
    > after a successful init.

    That doesn't solve the problem, because there are all kinds of
    circumstances where you have a "successful" init even when the
    user's init-file was not loaded. For example, many modular Emacs
    configurations will sustain errors in one or more modules using
    `condition-case', and report the errors as warnings for an
    improved debugging experience. We're back to the beginning.

    Besides, often one wants to test something in a plain Emacs but
    not in 'emacs -Q'. In this case, the call to `package-initialize'
    will still get inserted.

> I'd be very interested to work with maintainers of other package
> managers to see how we could make them better interoperate (e.g.
> make it possible to install with one tool, but activate&config with
> another).

Sounds like a great idea. However, do note that this will only be
useful for package managers that use the package.el format (e.g.
Quelpa, Cask, Pallet) and not for others (e.g. el-get, Borg,
straight.el).

Best,
Radon

[1]: https://www.reddit.com/r/emacs/comments/56fvgd/is_there_a_way_to_stop_emacs_from_adding_the/

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Stefan Monnier
> The problem is that even if I don't use package.el, there may be some
> stuff left in ~/.emacs.d/elpa from previous times.

Then don't do that: we want to make it as straightforward as possible for
users to install&use packages from GNU ELPA, so automatic activation of
those packages that were installed is a prerequisite.

Inevitably there will be situations where this design goal will clash
with the end-user who wants to use something else and will want to
explicitly "disable" package.el.

>> We could change tactic: only call package--ensure-init-file when the
>> user installs a package.
> I don't think I would enjoy this either.

Do you mean that it would be worse, or that it would be better but still
not good enough?  Based on your earlier message, I'm pretty sure I will
not come up with a solution which you really like, so I'm only aiming to
get "better" rather than "good enough".

> Would you consider it an acceptable solution to pop up a window or
> display a message telling the user that they should put
> `package-initialize' in their init-file, provided that we didn't have
> it get called during init?

No, I think this will reduce your annoyance level a tiny bit, along with
that of a handful of other users (all of whom likely already know
several different ways to work around the current annoyance), while
significantly increasing the annoyance level of many more new users who
just want to get on with their business without having to know anything
about how package.el works.

IOW, a bad trade-off.

>     The problem is that if there is an error while loading the
>     init-file, then the call to `package-initialize' still gets
>     inserted, if init hadn't gotten to the point where the actual call
>     to `package-initialize' was supposed to happen.

But if the call is only inserted during package-install it means that
you'd need both:
- use package-install
- in a session where your init failed to call package-initialize
for the problem to show up.  This combination seems a lot less likely
than the current one.

Another thing we could consider is to drop the automatic call to
package-initialize in lisp/startup.el (again, based on the idea that
this has now been made unnecessary by package--ensure-init-file).

>     Besides, often one wants to test something in a plain Emacs but
>     not in 'emacs -Q'. In this case, the call to `package-initialize'
>     will still get inserted.

I don't know what is a "plain Emacs but not in `emacs -Q`".  When I need
something like that I do something like

    mv ~/.emacs ~/tmp/
    <runemacs>
    mv ~/tmp/.emacs ~/

so any edits to ~/.emacs during the test will be happily thrown away
without bothering anyone.

So it seems like you're thinking of another kind of "plain Emacs but not
in `emacs -Q`".

>> I'd be very interested to work with maintainers of other package
>> managers to see how we could make them better interoperate (e.g.
>> make it possible to install with one tool, but activate&config with
>> another).
> Sounds like a great idea. However, do note that this will only be
> useful for package managers that use the package.el format (e.g.
> Quelpa, Cask, Pallet) and not for others (e.g. el-get, Borg,
> straight.el).

Part of "make them interoperate" may involve changing the formats
accepted&|used by the various tools, indeed (on-disk, on-the-server,
...).


        Stefan


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Noam Postavsky-2
In reply to this post by Stefan Monnier
On Sun, Aug 6, 2017 at 9:39 PM, Stefan Monnier <[hidden email]> wrote:
>
> Another thing is that rather then look for "(package-initialize)" in
> ~/.emacs we could keep track of whether package-initialize was called
> during initialization.  This will avoid the problem when the user placed
> his call in another file.

I believe we already do this in Emacs 26 (see Bug#24643/25819).

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Radon Rosborough
In reply to this post by Stefan Monnier
> > The problem is that even if I don't use package.el, there may be
> > some stuff left in ~/.emacs.d/elpa from previous times.
>
> Then don't do that:

I'm actually not sure what "that" is referring to here. Sorry if it is
obvious.

> we want to make it as straightforward as possible for users to
> install&use packages from GNU ELPA, so automatic activation of those
> packages that were installed is a prerequisite.

Agreed, but it doesn't have to be done in this way. We could
accomplish the same thing simply by providing a template init-file,
without any of the problems I've mentioned.

Is there any particular reason why providing a template init-file
would be a worse solution than modifying the init-file on the fly? If
so, could you explain to me what it is?

Let me reiterate: providing a template configuration file is an
*extremely common pattern*, and it's even common for the default
behavior of a program (when run with no configuration file) to be
generally considered unacceptable. The principle of least surprise
suggests that Emacs should use the same system as every other piece of
software with this kind of problem uses.

> Inevitably there will be situations where this design goal will
> clash with the end-user who wants to use something else and will
> want to explicitly "disable" package.el.

I'm fine with disabling package.el being an explicit step. Not with it
being an ongoing battle (where package.el strikes back every time I
accidentally use one of its functions without the proper advices
defined). I would be totally satisfied with there being package.el
code in the template init-file and no mention of alternative package
managers, as long as it ends with the template init-file (i.e. it's a
one-time thing).

> Do you mean that it would be worse, or that it would be better but
> still not good enough?

Better but still not good enough. I would consider modifying the
init-file automatically at startup as "atrocious" and doing it at
other times "undesirable but acceptable if it's really the only
solution".

> Based on your earlier message, I'm pretty sure I will not come up
> with a solution which you really like, so I'm only aiming to get
> "better" rather than "good enough".

That's OK. I certainly don't believe it's the responsibility of the
Emacs community to make me happy; sorry if I gave that implication.

> > Would you consider it an acceptable solution to pop up a window or
> > display a message telling the user that they should put
> > `package-initialize' in their init-file, provided that we didn't
> > have it get called during init?

> No,

Yeah, a popup window would be very annoying. It was just an offhand
suggestion. But consider that package.el doesn't currently display any
useful message in the echo area after it's finished (just whatever
byte-compilation message happened to be last). I don't think it would
be a degredation to user experience to have it say

    Finished installing 2 packages

and then if package-initialize wasn't called during init,

    Finished installing 2 packages, but please run
    M-x package-ensure-init-file to finalize installation

or something like that. Perhaps a *Warning* would be appropriate? As
long as the message conveyed a way to find out more details, and a
quick fix (I'm fine with advising to use a function that modifies the
init-file, as long as that's an intentional action).

I'm partially playing devil's advocate here; I really think the
template init-file is the way to go. But even if we do use the
template init-file, some better messaging would still be nice, I
think.

> (all of whom likely already know several different ways to work
> around the current annoyance)

Based on the Reddit thread I linked, most people who encounter this
problem don't know how to fix it. The ones who do spent an hour or two
figuring out how to do so because there's no documentation.

> But if the call is only inserted during package-install it means
> that you'd need both:
> - use package-install
> - in a session where your init failed to call package-initialize
> for the problem to show up. This combination seems a lot less likely
> than the current one.

This happened to me repeatedly when I was testing package managers,
since I obviously wanted to test installing packages in an environment
where they weren't already activated by my primary package manager. I
do agree that this situation is uncommon; it just strikes me that the
mechanism currently in use is rather fragile if it "breaks" in such a
situation.

> Another thing we could consider is to drop the automatic call to
> package-initialize in lisp/startup.el (again, based on the idea that
> this has now been made unnecessary by package--ensure-init-file).

I am strongly in support of this unless it means that we will be more
reluctant to eventually remove `package--ensure-init-file'. Almost all
of the tangible problems I mentioned in my original email stem from
package.el modifying the init-file on the fly; arguing for the
elimination of this behavior is therefore my highest priority.

> So it seems like you're thinking of another kind of "plain Emacs but
> not in `emacs -Q`".

No, I was thinking of the same situation. I know I have been bitten by
this before, but thinking back on it I can't say that any of my use
cases there are actually common in any way. Let's pretend I didn't
make that point; I don't think it is useful.

> >> I'd be very interested to work with maintainers of other package
> >> managers to see how we could make them better interoperate (e.g.
> >> make it possible to install with one tool, but activate&config with
> >> another).
> > Sounds like a great idea. However, do note that this will only be
> > useful for package managers that use the package.el format (e.g.
> > Quelpa, Cask, Pallet) and not for others (e.g. el-get, Borg,
> > straight.el).
>
> Part of "make them interoperate" may involve changing the formats
> accepted&|used by the various tools, indeed (on-disk, on-the-server,
> ...).

Note that source-based package managers such as Borg and straight.el
are fundamentally incompatible with the package.el format (or,
depending on your perspective, the package.el format is fundamentally
incompatible with source-based package managers such as Borg and
straight.el). el-get might benefit from this improved
interoperability, though it is an interesting case as it's the only
package manager I know of that uses both package.el and non-package.el
formats for packages.

Best,
Radon

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Mark Oteiza
In reply to this post by Stefan Monnier

Stefan Monnier <[hidden email]> writes:

>> Sure, I think it's reasonable for Emacs to provide special support for
>> packages which are built in. But there's such a thing as going too
>> far.  And I personally think that you've gone too far in providing
>> special support when that support actively makes it *more* difficult
>> to swap out an alternative implementation.
>
> You wrote lots and lots of lines of text just to complain about the
> addition of a single "(package-initialize)"

Not every user of package.el needs (package-initialize) in their init
file.  Not every user needs it at the beginning of the file, either.

> This said, the main motivation for calling package--ensure-init-file
> from package-initialize was to fix existing user's config where they had
> packages installed yet their .emacs didn't call package-initialize, so
> they had trouble configuring their packages.  One might argue that this
> situation is now mostly fixed and we could change tactic: only call
> package--ensure-init-file when the user installs a package.

The existing behaviour:

- subverts `package-load-list' settings
- breaks setting of package-archives that would normally happen before
  calling `package-initialize'
- writes init.el twice for some reason?
- does nothing to solve the disparity between packages and customize,
  which is the topic that opened the can of worms in the first place
- is basically undocumented

I can think of one other instance of a fragile init file lisp parser in
Emacs, and that one only exists for political reasons.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Colin Baxter
In reply to this post by Radon Rosborough
Dear Radon,

I would like to thank you for your extensive post. I found it very
useful and informative. This was not your motivation, I know, but it
went some way to confirm my own prejudice against using package
manager. I prefer to install external packages 'by hand'.

Best wishes,

Colin.

--
--
Colin Baxter
[hidden email]
GnuPG fingerprint: 68A8 799C 0230 16E7 BF68  2A27 BBFA 2492 91F5 41C8

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Stefan Monnier
In reply to this post by Mark Oteiza
> - subverts `package-load-list' settings

I don't know what this problem is.

> - breaks setting of package-archives that would normally happen before
>   calling `package-initialize'

How so?


        Stefan


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Stefan Monnier
In reply to this post by Radon Rosborough
>> > The problem is that even if I don't use package.el, there may be
>> > some stuff left in ~/.emacs.d/elpa from previous times.
>> Then don't do that:
> I'm actually not sure what "that" is referring to here.

"That" refers to leaving stuff in ~/.emacs.d/elpa when you're not using
package.el.

> Agreed, but it doesn't have to be done in this way. We could
> accomplish the same thing simply by providing a template init-file,
> without any of the problems I've mentioned.

Maybe we should provide a template init file.
[ This file should be basically empty (tho could contain lots of
comments.  ]
But auto-creating this file just because you start Emacs is also
a problem (similar to the auto-editing of this file you're complaining
about).

> Is there any particular reason why providing a template init-file
> would be a worse solution than modifying the init-file on the fly?

We also want to cater to old users of Emacs (who already have
their own ~/.emacs) who start to use package.el.

>> Inevitably there will be situations where this design goal will
>> clash with the end-user who wants to use something else and will
>> want to explicitly "disable" package.el.
> I'm fine with disabling package.el being an explicit step.  Not with it
> being an ongoing battle (where package.el strikes back every time I
> accidentally use one of its functions without the proper advices
> defined).

Which functions do you use accidentally?

>> Do you mean that it would be worse, or that it would be better but
>> still not good enough?
> Better but still not good enough.

Great.  Then let's try to do that.  I suggest you M-x report-emacs-bug
and request this change (so it gets a tracking number, and its
dedicated discussion).

> do agree that this situation is uncommon; it just strikes me that the
> mechanism currently in use is rather fragile if it "breaks" in such a
> situation.

I wonder what you mean by "break".

>> Another thing we could consider is to drop the automatic call to
>> package-initialize in lisp/startup.el (again, based on the idea that
>> this has now been made unnecessary by package--ensure-init-file).
> I am strongly in support of this

Great.  Then let's try to do that.  I suggest you M-x report-emacs-bug
and request this change (so it gets a tracking number, and its
dedicated discussion).

> unless it means that we will be more reluctant to eventually remove
> `package--ensure-init-file'.

I don't see why it would have such an effect.


        Stefan


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Interoperation between package managers (was: Friendly discussion about (package-initialize))

Stefan Monnier
In reply to this post by Radon Rosborough
>> Part of "make them interoperate" may involve changing the formats
>> accepted&|used by the various tools, indeed (on-disk, on-the-server,
>> ...).
> Note that source-based package managers such as Borg and straight.el
> are fundamentally incompatible with the package.el format (or,
> depending on your perspective, the package.el format is fundamentally
> incompatible with source-based package managers such as Borg and
> straight.el).

Hmm... really?  Could you expand on why that is?


        Stefan


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Radon Rosborough
In reply to this post by Stefan Monnier
> > > > The problem is that even if I don't use package.el, there may
> > > > be some stuff left in ~/.emacs.d/elpa from previous times.
> > >
> > > Then don't do that:
> >
> > I'm actually not sure what "that" is referring to here.
>
> "That" refers to leaving stuff in ~/.emacs.d/elpa when you're not
> using package.el.

As an end user, why should I have to know how package.el stores its
data in ~/.emacs.d? Isn't that an implementation detail? I'm concerned
that knowledge of this information is apparently required to prevent
package.el from taking actions all on its own (without being asked)
after you've switched to another package manager.

At a bare minimum, there needs to be a prominent notice about this in
the documentation [1]. But I still maintain that the entire situation
is very sketchy.

> Maybe we should provide a template init file.

I think this is a good idea not just to solve this problem with
package.el, but also to make it easier to nudge new users towards
Emacs configuration best practices without having to change defaults
(which, as we all know, is extremely contentious).

> [ This file should be basically empty (tho could contain lots of
> comments. ]

That can be debated later. At the minimum, though, it would have to
contain a (package-initialize), with lots of accompanying comments to
explain what it's doing there and where you should put your code to
configure packages.

> But auto-creating this file just because you start Emacs is also a
> problem (similar to the auto-editing of this file you're complaining
> about).

Agreed. However, it can't possibly be worse that the current
situation, right? Currently, a new file is created automatically,
*and* any existing one is modified. My proposal would limit this
behavior to automatic creation, and not modification.

Of course, the init-file shouldn't be created at startup in 'emacs -Q'
mode.

> We also want to cater to old users of Emacs (who already have their
> own ~/.emacs) who start to use package.el.

I do like it when things "just work", but there's only so much magic
you can do to that end before you start losing things like robustness,
predictability, and your sanity.

I know I've brought this up before, but can you name a single other
program that deals with the problem of bad default behavior by
automatically modifying a free-form user-written configuration file on
startup, depending on a number of undocumented heuristics? If nobody
else does it, I'm sure there's a reason for that.

I like to think that most everyone who uses Emacs is intelligent
enough to understand a simple concept like "the package manager must
be initialized before you can use the packages". The problem is that
this simple concept is made quite complicated by all the magic of
package.el, and it is not clearly documented at all. If we had better
communication with users, then there would be no need for magic.

And before you say that some people don't want to modify their
init-file, and would prefer to rely on interactive commands only, let
me point out that this problem only arises if you try to put package
configuration in your init-file.

> Which functions do you use accidentally?

I didn't mean that I use package.el functions accidentally. I mean
that I use package.el functions intentionally (when I am testing other
people's package managers, for example), but I accidentally forget to
define advices on internal package.el functions before doing so, with
the result that my init-file is modified for no reason and I have to
go and revert it.

> I wonder what you mean by "break".

I mean that it inserts a call to `package-initialize' in my init-file
when that call is both superfluous and actively harmful due to the
particulars of my setup. It objectively does the wrong thing in my
case; that is why I said "break". Not to imply that this isn't the
right thing to do in general, but it's definitely not always the right
thing to do.

> > > > > We could change tactic: only call package--ensure-init-file
> > > > > when the user installs a package.
> > > >
> > > > I don't think I would enjoy this either.
> > >
> > > Do you mean that it would be worse, or that it would be better but
> > > still not good enough?
> >
> > Better but still not good enough.
>
> Great. Then let's try to do that. I suggest you M-x report-emacs-bug
> and request this change

Will do, although I'll probably wait to see where this discussion goes
first, so I have the proper context.

> I don't see why it would have such an effect.

Because if we removed `package--ensure-init-file' subsequently, then
there would no longer be any automatic activation of packages during
init, except for new users who had a template init-file generated.

But I'm glad to hear that my concern was unfounded, and the removal of
init-file modification via `package--ensure-init-file' will not face
any more resistance than it does currently.

===== Something to consider =====

Let's say that one day, another package manager is accepted into Emacs
core. Users would have a choice between package.el and alternative.el,
whatever the latter may be. Does it still make sense for Emacs to
inject a whole bunch of magic into the init sequence, including
modifying the user's init-file without their permission under several
different conditions, catering specifically to package.el? Should
Emacs do this for both package managers? That doesn't make a lot of
sense, since of course the packages installed by the two might
conflict, and the user only needs one package manager. It's up to the
user to explicitly tell Emacs which package manager to use, by placing
(package-initialize) or (alternative-initialize) in their init-file.
That's the only reasonable solution; anything else is piling hacks on
top of hacks. Does this sound familiar?

===== An allegory =====

Let us travel to an alternate universe in which the snippet package
YASnippet was designed according to the same principles as package.el,
and it was included into Emacs core. What would this mean?

Well, firstly, there would be a function called `yas-initialize', and
you'd have to call it in order to load all the currently defined
snippets, and rebind <TAB> so that it would expand snippets if
appropriate.

Whenever you started Emacs, `yas-initialize' would be run
automatically, which would rebind <TAB> without asking you, and then
you'd get a comment inserted into your init-file:

    ;; Added by Yasnippet.  This must come before configurations of
    ;; installed snippets.  Don't delete this line.  If you don't want it,
    ;; just comment it out by adding a semicolon to the start of the line.
    ;; You may delete these explanatory comments.
    (yas-initialize)

You could prevent this by setting the variable `yas-enable-at-startup'
to nil, but this wouldn't help you in the case that your init-file
encountered an error somewhere, or any number of other interesting
edge cases. It would only take one time for that comment to get
inserted, and then Yasnippet would start getting activated at startup
again, even if you had previously tried to disable it.

Of course, being a highly capable Emacs user, you are very careful to
never encounter any of those situations... somehow. But one day, you
decide to try out Yasnippet, to see if you like it. You set up the
configuration just how you like it, in a sub-module of your
modularized init-file. But as soon as you load Yasnippet, that comment
is back again, and in the wrong place! After an hour or two of poring
through the source code, it looks like the only way to permanently
disable this behavior is to place an advice on an internal function.
You can hardly believe your eyes.

"Surely the Emacs developers didn't think that everyone needed
Yasnippet activated in just the same way, or even that everyone wants
to use Yasnippet, right? This must be a mistake. How could anybody
think that globally rebinding <TAB> on startup is a good idea?"

So you go and complain about the situation, only to be told things
like:

> Well, the rebinding of <TAB> won't make any difference if the user
> doesn't use Yasnippet (i.e., doesn't have any snippets defined).

> We want to make it as straightforward as possible for users to
> define and use snippets.

> People who object to this kind of thing are in the minority; it's
> more important to make the experience as seamless as possible for
> Yasnippet users.

> Just make sure to delete all your snippets whenever you're done
> using Yasnippet, that will solve the problem.

I'm being intentionally facetious here; I certainly don't intend to
give offense. I just think that looking at the situation from outside
the lens of package.el will lend some good perspective. After all, no
package (and even no package *manager*) other than package.el does
this kind of thing, and I've yet to hear a reason why package.el just
*has* to be different.

Best regards,
Radon

P.S. If you think that comparing package initialization to rebinding
<TAB> globally is unfair, I'd say on the contrary it's quite an
egalitarian comparison:

slow init time due to loading unneeded snippets
  <==>
slow init time due to loading duplicate packages

conflicts between <TAB> rebinding and user's bindings
  <==>
conflicts between stale package.el packages and user's packages

unpredictable behavior of <TAB> because installed snippets are not
synced between machines
  <==>
unpredictable behavior of Emacs config because installed packages are
not synced between machines

disrespect of the user's desire to use another snippet manager
  <==>
disrespect of the user's desire to use another package manager

P.P.S. I originally started this comparison as a joke, but I'd
honestly be less offended by the <TAB> thing.

[1]: https://www.gnu.org/software/emacs/manual/html_node/emacs/Package-Files.html

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Interoperation between package managers (was: Friendly discussion about (package-initialize))

Radon Rosborough
In reply to this post by Stefan Monnier
> > > Part of "make them interoperate" may involve changing the formats
> > > accepted&|used by the various tools, indeed (on-disk,
> > > on-the-server, ...).
> >
> > Note that source-based package managers such as Borg and straight.el
> > are fundamentally incompatible with the package.el format (or,
> > depending on your perspective, the package.el format is
> > fundamentally incompatible with source-based package managers such
> > as Borg and straight.el).
>
> Hmm... really?  Could you expand on why that is?

Perhaps I should not have said the package.el format is fundamentally
incompatible with source-based package managers. What I should have
said is that source-based package managers would gain little by being
compatible with the package.el format. This is because the package.el
format is inherently unable to provide for the basic features that
define a source-based package manager.

In particular, packages that are installed via package.el cannot be
version-controlled, since the package.el format dictates a flat
collection of *.el (and other) files, with no version-control
information attached other than a version number. As a corollary, it
is impossible to use the package.el format to make local modifications
to a package in a controlled way, or to contribute changes upstream
using package.el.

All of these actions (version-control operations, local changes,
upstream contributions) are foundational to a source-based package
manager. The only way to achieve them is to run packages directly from
their VCS repositories (Borg, see [1]) or via symlinks that point at
said repositories (straight.el, see [2]). Thus, from a purely
technical standpoint, it is not useful for a source-based package
manager to interoperate with package.el in any meaningful way.

I do not wish to say that package.el is flawed for failing to provide
these features, since package.el is designed to work in the absence of
a local VCS installed, and it is impossible to provide these features
without making that assumption. But for package managers which do make
that assumption, the package.el format (in my opinion) is no longer as
relevant.

Best,
Radon

[1]: https://github.com/emacscollective/borg
[2]: https://github.com/raxod502/straight.el

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Eli Zaretskii
In reply to this post by Radon Rosborough
> From: Radon Rosborough <[hidden email]>
> Date: Wed, 9 Aug 2017 20:32:59 -0700
> Cc: [hidden email]
>
> > [ This file should be basically empty (tho could contain lots of
> > comments. ]
>
> That can be debated later. At the minimum, though, it would have to
> contain a (package-initialize), with lots of accompanying comments to
> explain what it's doing there and where you should put your code to
> configure packages.

Can someone explain, preferably in concise form, why are we having
this discussion about in which file to have the call to
package-initialize, given that startup.el already calls that function
(as IMO it should)?

As you might guess, I don't use package.el, but this issue, which to
my opinion seems to be already solved The Right Way, continues to
generate such prolonged discussions that I wonder where did we make
the wrong turn with package.el.

Thanks.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Radon Rosborough
> Can someone explain, preferably in concise form, why are we having
> this discussion about in which file to have the call to
> package-initialize

Gladly.

Currently, Emacs has the opinion that you should call
`package-initialize' in your init-file, and it enforces this opinion
by automatically inserting such a call directly into your init-file at
startup, or whenever you initialize the package management system.

I view this behavior as wrong. Other people feel that it is the best
solution to the problem of users putting package configuration in
their init-file, which runs before `package-initialize' is called in
startup.el.

That is why we are having this discussion.

> startup.el already calls that function (as IMO it should)

I disagree, but won't speak further unless you think it will be
relevant to the discussion.

> where did we make the wrong turn with package.el.

When Emacs started modifying the init-file specifically for package.el.

Best,
Radon

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Eli Zaretskii
> From: Radon Rosborough <[hidden email]>
> Date: Wed, 9 Aug 2017 21:39:44 -0700
> Cc: Stefan Monnier <[hidden email]>, [hidden email]
>
> Currently, Emacs has the opinion that you should call
> `package-initialize' in your init-file, and it enforces this opinion
> by automatically inserting such a call directly into your init-file at
> startup, or whenever you initialize the package management system.
>
> I view this behavior as wrong. Other people feel that it is the best
> solution to the problem of users putting package configuration in
> their init-file, which runs before `package-initialize' is called in
> startup.el.
>
> That is why we are having this discussion.
>
> > startup.el already calls that function (as IMO it should)
>
> I disagree, but won't speak further unless you think it will be
> relevant to the discussion.

This sounds like a contradiction.  If we call package-initialize from
startup.el, why does it have to also be called from the init files?
And since you are opposed to Emacs putting tyhis call to init files,
why do you disagree that calling package-initialize in startup.el is
wrong?  After all, that's where Emacs performs all the initialization
for the upcoming session, and that should include initializing the
packages which need initialization.

> > where did we make the wrong turn with package.el.
>
> When Emacs started modifying the init-file specifically for package.el.

So having the call in startup.el is correct after all?  Now I'm
utterly confused.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Radon Rosborough
> This sounds like a contradiction. [...] Now I'm utterly confused.

Sorry, it looks like I failed to provide enough context in the name of
being concise. Let me try again.

The issue is complicated because there are actually a number of
different issues at play, which all interact in different ways.

Firstly, let me describe my ideal world. This would be a world in
which `package-initialize' was not called automatically anywhere, or
inserted into the init-file, or anything. However, new users could
still benefit from package.el because Emacs would generate a template
init-file which would contain a call to `package-initialize', if no
init-file already existed.

Now there is also the question of where it is appropriate to call
`package-initialize'. IMO, the only appropriate place to call it is in
the user's init-file. Doing it anywhere else smells like unnecessary
magic, and limits customizability. For example, calling it in
startup.el after loading the init-file means that package
customizations cannot be put in the init-file (unless you use
`after-init-hook', an advanced and rather nonstandard approach), but
the package management system still works after init. Can you possibly
think of any setup that would be *more* confusing to new users?

Then there is the issue of Emacs automatically modifying the init-file
to stick a call to `package-initialize' in it. I think this behavior
is disgusting because of how fragile and nonstandard it is.

Now let me address your specific concerns.

> If we call package-initialize from startup.el, why does it have to
> also be called from the init files?

IMO, it should not be called anywhere. I think having it only called
in startup.el would be a reasonable compromise. Having Emacs insist on
putting the call into the init-file, but then *also* calling it in
startup.el, makes no sense.

> since you are opposed to Emacs putting tyhis call to init files, why
> do you disagree that calling package-initialize in startup.el is
> wrong?

I do think that the correct place to call `package-initialize' is in
the init-file, but I object to Emacs taking care of this
automatically. It might seem like the best solution at first glance,
but doing it this way has a laundry list of horrible side effects (see
my original email).

> that's where Emacs performs all the initialization for the upcoming
> session, and that should include initializing the packages which
> need initialization.

Like I said, having the call be only in startup.el would be a
perfectly reasonable compromise IMO. My main issue is with Emacs
modifying the init-file automatically.

Does this make more sense? I can elaborate, but briefly:

1. I want `package--ensure-init-file' to go away ASAP.
2. I think Emacs should generate a template init-file; in addition to
   making `package--ensure-init-file' mostly superfluous, this would
   have other advantages in terms of improving default user
   experience.
3. If possible, I'd like startup.el to not call `package-initialize'
   either, but this is an order of magnitude less important than (1)
   and (2).

Best,
Radon

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Eli Zaretskii
> From: Radon Rosborough <[hidden email]>
> Date: Thu, 10 Aug 2017 10:06:34 -0700
> Cc: Stefan Monnier <[hidden email]>, [hidden email]
>
> Now there is also the question of where it is appropriate to call
> `package-initialize'. IMO, the only appropriate place to call it is in
> the user's init-file. Doing it anywhere else smells like unnecessary
> magic, and limits customizability. For example, calling it in
> startup.el after loading the init-file means that package
> customizations cannot be put in the init-file (unless you use
> `after-init-hook', an advanced and rather nonstandard approach), but
> the package management system still works after init. Can you possibly
> think of any setup that would be *more* confusing to new users?

I'm probably missing some details here, but in principle I don't see
here anything that should be confusing or hard to get right.  We do
that with other tricky stuff, like customizations of the basic faces,
which we read from .emacs after Emacs already started and displayed
its first frame.  And using hooks is not such scary stuff for new
users, either.  To say nothing of the fact that new users aren't
expected to mess with this anyway, they should just use what package
installation procedure arranged for.

> > If we call package-initialize from startup.el, why does it have to
> > also be called from the init files?
>
> IMO, it should not be called anywhere. I think having it only called
> in startup.el would be a reasonable compromise. Having Emacs insist on
> putting the call into the init-file, but then *also* calling it in
> startup.el, makes no sense.

Thanks, but that doesn't really answer my question.  I asked why do we
put a call to package-initialize into user init file when we already
have that very call in startup.el.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Radon Rosborough
> I'm probably missing some details here

I'm missing some details too, since I don't use the Custom system, and
that seems to have been part of this discussion in the past. I expect
that the fact there are so many details is what makes this discussion
keep coming up, because nobody's solution caters to every use case.

> using hooks is not such scary stuff for new users

Fine, but do we really want to tell users to put the entirety of their
init-file inside `after-init-hook'? That seems like an anti-pattern to
me. It's either `package-initialize' which has to move, or the entire
rest of the init-file. It makes more sense to me to move
`package-initialize'.

> new users aren't expected to mess with this anyway

So if we implemented a template init-file, then new users who didn't
have a init-file previously would not need to do anything.
Furthermore, existing users who are comfortable with customizing Emacs
would not have a problem either, since the concept of "you must
initialize the package management system before you can use packages"
is trivially simple. The only people disadvantaged are new users who
already had an existing init-file.

If we keep calling `package-initialize' in startup.el, then things
will work as expected for these users as well, unless they happen to
manually write some customizations of packages into their init-file.
But now the set of users who are disadvantaged is limited to:

1. new users
2. who aren't comfortable with modifying their init-file
3. who nevertheless had an existing init-file
4. and nevertheless add Lisp code directly to their init-file anyway

In my opinion, this is a rather narrow intersection, and the problems
introduced by automatic modification of the init-file far outweigh the
inconvenience to this group of users.

> that doesn't really answer my question. I asked why do we put a call
> to package-initialize into user init file when we already have that
> very call in startup.el.

You'll have to ask Stefan. I am the one arguing that this behavior
makes no sense and should be eliminated ASAP.

In short, however, this measure was introduced in an effort to allow
for package customizations to be put in the init-file without
`package-initialize' also being put in the init-file (by the user).

best,
Radon

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly discussion about (package-initialize)

Mark Oteiza
In reply to this post by Eli Zaretskii

Eli Zaretskii <[hidden email]> writes:

>> > If we call package-initialize from startup.el, why does it have to
>> > also be called from the init files?
>>
>> IMO, it should not be called anywhere. I think having it only called
>> in startup.el would be a reasonable compromise. Having Emacs insist on
>> putting the call into the init-file, but then *also* calling it in
>> startup.el, makes no sense.
>
> Thanks, but that doesn't really answer my question.  I asked why do we
> put a call to package-initialize into user init file when we already
> have that very call in startup.el.

The form is put there in order to coddle users who copy-paste
configurations a) into their init files, or b) in Customize settings,
that use symbols from packages before the packages are (auto-)loaded.

To quote the implementer (from this thread[2]):

> Another consequence is that users can't paste
> configurations into their init file in the way that 90% of the
> (emacs-related) internet is telling them to.
> By adding the chosen snippet to the user's init file we are preventing
> this sort of problem for new users, instead of telling them it's their
> fault for not understanding package.el.

FWIW here are some other links to threads surrounding the issue when it
was being debated/implemented:

[0] https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg00897.html
[1] https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html
[2] https://lists.gnu.org/archive/html/emacs-devel/2015-04/msg00002.html

12
Loading...