EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Pierre Neidhardt

I've recently hacked around EWW and since it's coming into good shape
I'm thinking of committing this upstream.

Before sending a patch, here is the current state of my hacks:

        https://github.com/Ambrevar/dotfiles/blob/master/.emacs.d/lisp/init-eww.el

Summary:

- Add `eww-copy-page-title' that mirrors `eww-copy-page-url', it's
  useful enough.

- Enhances eww-next-url / eww-previous-url so that when there is no next /
  previous links, it tries to increment / decrement the last number in
  the last element of the URL.  ("do what you mean" style.)

- Add a `eww-reload-all' command to reload all *eww* buffers, it's
  useful when using desktop-mode but when `eww-restore-desktop' is nil.

- Make `eww' so that the default value is in the prompt, ready to be
  edited by the user.  Very convenient in my opinion.

- Change `eww-open-in-new-buffer' so that it queries for a URL instead
  of cloning the current buffer (which is not very useful in my
  opinion).

- Make eww-update-header-line-format also update the buffer name so that
  it contains the page title.  _Very useful_ to browse / search *eww*
  buffers (think Ivy/Helm).

- Ask for tags when saving a bookmark.  Tags are stored under the key
  :tags as a list of strings.

- Make `eww-add-bookmark' run a customizable function to decide when.
  to error out.  For instance, error out when a duplicate is detected
  with protocol stripped out (https://foo.bar is seen as a duplicate of
  http://foo.bar).

- Make `eww-write-bookmarks' run a customizable function before saving
  the file.  That function can be used, for instance, to detect
  duplicates or to sort the bookmarks.  This would make the eww-bookmarks
  file more friendly to versioning.

- Bookmarks can have a mark which is a string saved under the key :mark.
  The mark should be unique.  It could be used like the "quickmark"
  function found in some browsers: use it to quickly load a
  bookmark. (Work in progress.)

- Bookmarks can have a search engine which is either appended to the
  bookmark' URL if it does not start with "https?://", or used as-is
  otherwise.
  The search engine is stored as a string under the key :search.
  A "%s" must be present in the search engine string as a place-holder
  for the query.

- Make `eww-bookmark-prepare' only load  bookmarks from file if not
  already set.  This makes it possible to display a custom / narrowed
  list of bookmarks in the bookmark buffer.

- Make `eww-bookmark-prepare' display the mark, the tags and the search
  engine, if available.  Work in progress.  I'm thinking of using a
  different face for the mark if a search engine is present.

- Add a `eww-bookmarks-by-tags' command which queries the user for a
  completing list of tags and then displays a bookmark buffers of all
  the bookmarks which match the tags.  The matching can be either
  inclusive or exclusive (bookmarks which match at least one tag vs. all
  of them).

- Make `eww--dwim-expand-url' follow a different logic to bind it all together:

  - With a multi-word query, if first word is a mark of a bookmark with a search engine,
  then use the said search engine over the rest of the query.

  - With a single word query, if first word is a mark then open the
    corresponding bookmark.

  - Else query the default search engine.

- Fix `eww-forward-url' as it seems to corrupt the history.  (Work in progress.)

Of course in its present state my hacks are what they are, very hacky.
It needs to be made more customizable and interfaceable.

What do you think?

--
Pierre Neidhardt

Power corrupts.  Absolute power is kind of neat.
                -- John Lehman, Secretary of the Navy, 1981-1987

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Lars Ingebrigtsen
Pierre Neidhardt <[hidden email]> writes:

> I've recently hacked around EWW and since it's coming into good shape
> I'm thinking of committing this upstream.
>
> Before sending a patch, here is the current state of my hacks:

Please send separate patches for each feature.  :-)

> - Add `eww-copy-page-title' that mirrors `eww-copy-page-url', it's
>   useful enough.

Yup.

> - Enhances eww-next-url / eww-previous-url so that when there is no next /
>   previous links, it tries to increment / decrement the last number in
>   the last element of the URL.  ("do what you mean" style.)

Hm...  Are there many web sites where that is meaningful?

> - Add a `eww-reload-all' command to reload all *eww* buffers, it's
>   useful when using desktop-mode but when `eww-restore-desktop' is nil.

Sure.

> - Make `eww' so that the default value is in the prompt, ready to be
>   edited by the user.  Very convenient in my opinion.

No, that's against Emacs' convention for prompting.  The default is
always stashed in `M-n'.

> - Change `eww-open-in-new-buffer' so that it queries for a URL instead
>   of cloning the current buffer (which is not very useful in my
>   opinion).

That's not what it's supposed to do -- it opens the link under point in
a new buffer.

> - Make eww-update-header-line-format also update the buffer name so that
>   it contains the page title.  _Very useful_ to browse / search *eww*
>   buffers (think Ivy/Helm).

Sounds nice, but also sounds like something not everybody would want, so
it should have a configuration option to switch on/off (but can default
to on).

> - Ask for tags when saving a bookmark.  Tags are stored under the key
>   :tags as a list of strings.

Good idea.

> - Make `eww-add-bookmark' run a customizable function to decide when.
>   to error out.  For instance, error out when a duplicate is detected
>   with protocol stripped out (https://foo.bar is seen as a duplicate of
>   http://foo.bar).

Hm...  Doesn't really sound all that useful?  But having that command
give feedback (and perhaps query the user about what to do) in that
situation would be handy.

> - Make `eww-write-bookmarks' run a customizable function before saving
>   the file.  That function can be used, for instance, to detect
>   duplicates or to sort the bookmarks.  This would make the eww-bookmarks
>   file more friendly to versioning.

Sure.

> - Bookmarks can have a mark which is a string saved under the key :mark.
>   The mark should be unique.  It could be used like the "quickmark"
>   function found in some browsers: use it to quickly load a
>   bookmark. (Work in progress.)

A mark in addition to a tag?  Sounds like a bit more than most users
would want to invest in a bookmarking system, but I don't object.

> - Bookmarks can have a search engine which is either appended to the
>   bookmark' URL if it does not start with "https?://", or used as-is
>   otherwise.
>   The search engine is stored as a string under the key :search.
>   A "%s" must be present in the search engine string as a place-holder
>   for the query.

Hm.  Sounds more complicated than users will want to do to me.

> - Make `eww-bookmark-prepare' only load  bookmarks from file if not
>   already set.  This makes it possible to display a custom / narrowed
>   list of bookmarks in the bookmark buffer.

I don't quite follow...  What about just adding narrowing and sorting to
that mode?

> - Make `eww-bookmark-prepare' display the mark, the tags and the search
>   engine, if available.  Work in progress.  I'm thinking of using a
>   different face for the mark if a search engine is present.

Sure.

> - Add a `eww-bookmarks-by-tags' command which queries the user for a
>   completing list of tags and then displays a bookmark buffers of all
>   the bookmarks which match the tags.  The matching can be either
>   inclusive or exclusive (bookmarks which match at least one tag vs. all
>   of them).

Sounds nice.

> - Make `eww--dwim-expand-url' follow a different logic to bind it all together:
>
>   - With a multi-word query, if first word is a mark of a bookmark with a search engine,
>   then use the said search engine over the rest of the query.
>
>   - With a single word query, if first word is a mark then open the
>     corresponding bookmark.
>
>   - Else query the default search engine.

Sounds confusing.  :-)  But also quite DWIM, which I like.

> - Fix `eww-forward-url' as it seems to corrupt the history.  (Work in progress.)
>
> Of course in its present state my hacks are what they are, very hacky.
> It needs to be made more customizable and interfaceable.
>
> What do you think?

Sounds great to me.  :-)  Make each thing into a patch (with
documentation) and let the apply-to-Emacs-fest commence.  That is, if
you've been through the assign-copyright-to-the-FSF-process.  I don't
see you in the copyright.list file, but that's apparently out of date
these days...

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Pierre Neidhardt

Lars Ingebrigtsen <[hidden email]> writes:

>> - Enhances eww-next-url / eww-previous-url so that when there is no next /
>>   previous links, it tries to increment / decrement the last number in
>>   the last element of the URL.  ("do what you mean" style.)
>
> Hm...  Are there many web sites where that is meaningful?

Emacs (and all GNU) mailing list archives for a start! :)
Many websites use increments in URL while they forget the previous/next
hint.

>> - Change `eww-open-in-new-buffer' so that it queries for a URL instead
>>   of cloning the current buffer (which is not very useful in my
>>   opinion).
>
> That's not what it's supposed to do -- it opens the link under point in
> a new buffer.

Sorry, I meant when there is no link under point.
Right now there is no way to just open a new eww buffer.

>> - Make eww-update-header-line-format also update the buffer name so that
>>   it contains the page title.  _Very useful_ to browse / search *eww*
>>   buffers (think Ivy/Helm).
>
> Sounds nice, but also sounds like something not everybody would want, so
> it should have a configuration option to switch on/off (but can default
> to on).

Sure.

>> - Make `eww-add-bookmark' run a customizable function to decide when.
>>   to error out.  For instance, error out when a duplicate is detected
>>   with protocol stripped out (https://foo.bar is seen as a duplicate of
>>   http://foo.bar).
>
> Hm...  Doesn't really sound all that useful?  But having that command
> give feedback (and perhaps query the user about what to do) in that
> situation would be handy.

Why not useful?  `eww-add-bookmark' already does duplicate detection,
except that it's very dumb: it won't realize if two bookmarks are the
same up to the protocol.

The command would obviously explain what's wrong when erroring out.

>> - Bookmarks can have a mark which is a string saved under the key :mark.
>>   The mark should be unique.  It could be used like the "quickmark"
>>   function found in some browsers: use it to quickly load a
>>   bookmark. (Work in progress.)
>
> A mark in addition to a tag?  Sounds like a bit more than most users
> would want to invest in a bookmarking system, but I don't object.

Maybe I should have explained the overall bookmark logic beforehand :p

- "tags" are optional words used to categorize bookmarks, e.g. "cinema", "emacs",
  "books".  They can be used to filter & lookup bookmarks.

- "mark" serves two purposes: it allows to open a link with a simple
  keybinding (optional) + it serves as a prefix for search engines (in
  which case it's no longer optional).  For example, say github has mark
  "gh", then

                M-x eww RET gh

  opens github, so does `C-c e g h' if `C-c e' is quickmark prefix key.
  If github comes with a search engine `:search "/search?q=%s", then

                M-x eww RET gh foobar

  opens a github search query of "foobar".

The point of centralizing search engines, quickmarks and bookmarks is
that it makes it easier to configure and, most importantly, it allows
for searching for all URLs at the same spot (name the bookmarks).

>> - Bookmarks can have a search engine which is either appended to the
>>   bookmark' URL if it does not start with "https?://", or used as-is
>>   otherwise.
>>   The search engine is stored as a string under the key :search.
>>   A "%s" must be present in the search engine string as a place-holder
>>   for the query.
>
> Hm.  Sounds more complicated than users will want to do to me.

See my previous comment.

I think this is actually simpler that adding search engine support with
seperate functions and variables.  With my suggestion we re-use the
current implementation without adding any new variable / function.

>> - Make `eww-bookmark-prepare' only load  bookmarks from file if not
>>   already set.  This makes it possible to display a custom / narrowed
>>   list of bookmarks in the bookmark buffer.
>
> I don't quite follow...  What about just adding narrowing and sorting to
> that mode?

We could also do, but that does not compose as well as my suggestion.
For instance, how would you filter bookmarks by tags?
Narrowind and sorting would not be enough.

>> - Make `eww--dwim-expand-url' follow a different logic to bind it all together:
>>
>>   - With a multi-word query, if first word is a mark of a bookmark with a search engine,
>>   then use the said search engine over the rest of the query.
>>
>>   - With a single word query, if first word is a mark then open the
>>     corresponding bookmark.
>>
>>   - Else query the default search engine.
>
> Sounds confusing.  :-)  But also quite DWIM, which I like.
The description is, but in practice it's crystal clear and always does
what you want.

> Sounds great to me.  :-)  Make each thing into a patch (with
> documentation) and let the apply-to-Emacs-fest commence.  That is, if
> you've been through the assign-copyright-to-the-FSF-process.  I don't
> see you in the copyright.list file, but that's apparently out of date
> these days...

I am assigned to Emms, does it count?

--
Pierre Neidhardt


signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Lars Ingebrigtsen
Pierre Neidhardt <[hidden email]> writes:

>>> - Enhances eww-next-url / eww-previous-url so that when there is no next /
>>>   previous links, it tries to increment / decrement the last number in
>>>   the last element of the URL.  ("do what you mean" style.)
>>
>> Hm...  Are there many web sites where that is meaningful?
>
> Emacs (and all GNU) mailing list archives for a start! :)
> Many websites use increments in URL while they forget the previous/next
> hint.

Hm...  The numbers in the URLs often (usually) don't refer to the
next/previous article, but are a global thing that count upwards.

But perhaps it'll work out nice on many web sites.  I'm not against
adding this and see how it works out.  If it turns out to be less than
helpful, we can remove it again.

>>> - Change `eww-open-in-new-buffer' so that it queries for a URL instead
>>>   of cloning the current buffer (which is not very useful in my
>>>   opinion).
>>
>> That's not what it's supposed to do -- it opens the link under point in
>> a new buffer.
>
> Sorry, I meant when there is no link under point.
> Right now there is no way to just open a new eww buffer.

Oh, I see.  Yes, that's a good idea.

>>> - Make `eww-add-bookmark' run a customizable function to decide when.
>>>   to error out.  For instance, error out when a duplicate is detected
>>>   with protocol stripped out (https://foo.bar is seen as a duplicate of
>>>   http://foo.bar).
>>
>> Hm...  Doesn't really sound all that useful?  But having that command
>> give feedback (and perhaps query the user about what to do) in that
>> situation would be handy.
>
> Why not useful?

Because the vast majority of users won't create such a function.  It's
more useful (i.e., for more people) if things work the way it should be
default.

> `eww-add-bookmark' already does duplicate detection, except that it's
> very dumb: it won't realize if two bookmarks are the same up to the
> protocol.

But then it should be smarter, I think.  :-)

>> A mark in addition to a tag?  Sounds like a bit more than most users
>> would want to invest in a bookmarking system, but I don't object.
>
> Maybe I should have explained the overall bookmark logic beforehand :p
>
> - "tags" are optional words used to categorize bookmarks, e.g. "cinema", "emacs",
>   "books".  They can be used to filter & lookup bookmarks.
>
> - "mark" serves two purposes: it allows to open a link with a simple
>   keybinding (optional) + it serves as a prefix for search engines (in
>   which case it's no longer optional).  For example, say github has mark
>   "gh", then
>
> M-x eww RET gh
>
>   opens github, so does `C-c e g h' if `C-c e' is quickmark prefix key.
>   If github comes with a search engine `:search "/search?q=%s", then
>
> M-x eww RET gh foobar
>
>   opens a github search query of "foobar".

Ah, I see.  Yes, that sounds very nice and useful.

>>> - Make `eww-bookmark-prepare' only load  bookmarks from file if not
>>>   already set.  This makes it possible to display a custom / narrowed
>>>   list of bookmarks in the bookmark buffer.
>>
>> I don't quite follow...  What about just adding narrowing and sorting to
>> that mode?
>
> We could also do, but that does not compose as well as my suggestion.
> For instance, how would you filter bookmarks by tags?

Add a "narrow to tag" command to the mode?

>> Sounds great to me.  :-)  Make each thing into a patch (with
>> documentation) and let the apply-to-Emacs-fest commence.  That is, if
>> you've been through the assign-copyright-to-the-FSF-process.  I don't
>> see you in the copyright.list file, but that's apparently out of date
>> these days...
>
> I am assigned to Emms, does it count?

Hm...  If it's specific to Emms, I don't think it's enough?  Anybody
know?

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Pierre Neidhardt

Lars Ingebrigtsen <[hidden email]> writes:

>>>> - Make `eww-add-bookmark' run a customizable function to decide when.
>>>>   to error out.  For instance, error out when a duplicate is detected
>>>>   with protocol stripped out (https://foo.bar is seen as a duplicate of
>>>>   http://foo.bar).
>>>
>>> Hm...  Doesn't really sound all that useful?  But having that command
>>> give feedback (and perhaps query the user about what to do) in that
>>> situation would be handy.
>>
>> Why not useful?
>
> Because the vast majority of users won't create such a function.  It's
> more useful (i.e., for more people) if things work the way it should be
> default.
>
>> `eww-add-bookmark' already does duplicate detection, except that it's
>> very dumb: it won't realize if two bookmarks are the same up to the
>> protocol.
>
> But then it should be smarter, I think.  :-)
Sorry if I wasn't clear: I would not only make it customizable but also
set the default to a smart function, so that both `eww-write-bookmark'
and `eww-add-bookmark' exhibit a smarter behaviour.  If that would not
be smart enough to the users, they could still customize it.

> We could also do, but that does not compose as well as my suggestion.
>> For instance, how would you filter bookmarks by tags?
>
> Add a "narrow to tag" command to the mode?

I have to think about it a little more.

>>> Sounds great to me.  :-)  Make each thing into a patch (with
>>> documentation) and let the apply-to-Emacs-fest commence.  That is, if
>>> you've been through the assign-copyright-to-the-FSF-process.  I don't
>>> see you in the copyright.list file, but that's apparently out of date
>>> these days...
>>
>> I am assigned to Emms, does it count?
>
> Hm...  If it's specific to Emms, I don't think it's enough?  Anybody
> know?
I can sign another assignment, no problem with that :)

--
Pierre Neidhardt

It will be advantageous to cross the great stream ... the Dragon is on
the wing in the Sky ... the Great Man rouses himself to his Work.

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Lars Ingebrigtsen
Pierre Neidhardt <[hidden email]> writes:

> Sorry if I wasn't clear: I would not only make it customizable but also
> set the default to a smart function, so that both `eww-write-bookmark'
> and `eww-add-bookmark' exhibit a smarter behaviour.  If that would not
> be smart enough to the users, they could still customize it.

Oh, OK.  Sounds good to me.

> I can sign another assignment, no problem with that :)

Great; I'll send you the form off-list.

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Lars Ingebrigtsen
In reply to this post by Pierre Neidhardt
Pierre Neidhardt <[hidden email]> writes:

> I can sign another assignment, no problem with that :)

Uhm, I can't seem to find the assignment template now...  Can somebody
else send it to Pierre?

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Eli Zaretskii
> From: Lars Ingebrigtsen <[hidden email]>
> Date: Mon, 16 Apr 2018 14:32:11 +0200
> Cc: [hidden email]
>
> Pierre Neidhardt <[hidden email]> writes:
>
> > I can sign another assignment, no problem with that :)
>
> Uhm, I can't seem to find the assignment template now...  Can somebody
> else send it to Pierre?

Sent off-list.

Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Charles A. Roelli
In reply to this post by Pierre Neidhardt
> From: Pierre Neidhardt <[hidden email]>
> Date: Mon, 16 Apr 2018 15:56:40 +0530
>
> I've recently hacked around EWW and since it's coming into good shape
> I'm thinking of committing this upstream.
>
> Before sending a patch, here is the current state of my hacks:
>
> https://github.com/Ambrevar/dotfiles/blob/master/.emacs.d/lisp/init-eww.el
>
> Summary:
>
> - ...

BTW, you may also be interested in Bookmark+'s support for EWW, which
includes:

  - changing EWW to use standard Emacs bookmarks instead of EWW's own
    implementation (so you get the annotation and position-saving
    features for free!),
  - automatic renaming of EWW buffers based on their URL or page title
    (not switched on by default),

  - and more generically (for every bookmark type under the sun),
    - tagging and searching bookmarks via these tags,
    - and dynamically sorting the bookmark list.

The Bookmark+ code specifically for EWW is at:

  https://www.emacswiki.org/emacs/bookmark%2b-1.el

I don't know how interested people here would be in integrating
Bookmark+ (or just its EWW support) into Emacs itself, but it may be
worth looking at.

Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Lars Ingebrigtsen
In reply to this post by Pierre Neidhardt
Pierre Neidhardt <[hidden email]> writes:

> - Enhances eww-next-url / eww-previous-url so that when there is no next /
>   previous links, it tries to increment / decrement the last number in
>   the last element of the URL.  ("do what you mean" style.)

Apropos of this, have you by any chance read this news item?

http://www.cbc.ca/news/canada/nova-scotia/freedom-of-information-request-privacy-breach-teen-speaks-out-1.4621970

This teenager was put in jail for decrementing the number in an URL
and downloading the contents.  Will this functionality open up all eww
users for prosecution?  :-)

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Reply | Threaded
Open this post in threaded view
|

Re: EWW improvements: open in new buffer, tags, quickmarks, search engines, ...

Pierre Neidhardt
In reply to this post by Charles A. Roelli

Charles A. Roelli <[hidden email]> writes:

> BTW, you may also be interested in Bookmark+'s support for EWW, which
> includes:
>
>   - changing EWW to use standard Emacs bookmarks instead of EWW's own
>     implementation (so you get the annotation and position-saving
>     features for free!),
>   - automatic renaming of EWW buffers based on their URL or page title
>     (not switched on by default),
>
>   - and more generically (for every bookmark type under the sun),
>     - tagging and searching bookmarks via these tags,
>     - and dynamically sorting the bookmark list.
>
> The Bookmark+ code specifically for EWW is at:
>
>   https://www.emacswiki.org/emacs/bookmark%2b-1.el
>
> I don't know how interested people here would be in integrating
> Bookmark+ (or just its EWW support) into Emacs itself, but it may be
> worth looking at.
Very good point, thanks for bringing it up!  I'll see if it's possible
to embed the mark and the search engine then.

--
Pierre Neidhardt

signature.asc (497 bytes) Download Attachment