Christmas wish: Literate Elisp

classic Classic list List threaded Threaded
44 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Christmas wish: Literate Elisp

arthur miller
Hello,

I have a question/proposal about Elisp and literate programming.
Well, more of a proposal then a question, but here it is:

From Wikipedia: "Literate programming is a programming paradigm
introduced by Donald Knuth in which a computer program is given
an explanation of its logic in a natural language, such as English,
interspersed with snippets of macros and traditional source code,
from which compilable source code can be generated."

Emacs already supports a form of literal programming in form of
org mode and babel where we can insert code for programming
languages in-between #+BEGIN_SRC and #+END_SRC markers,
which is super nice and cool feature.

However I got a thought that LISPs (lisp-like languages), have natural
code markers, since all code is enclosed with parenthesis. Thus one
could see '(' and ')' as code markers in literate-programming style,
more of as Knuth proposed. In other words, LISP (or at least Elisp)
does not need special markers to denote start and end of code. Unlike
Haskell, there is no need to use '\begin_code' or '>' to differentiate
code from text (comments).

My proposal is to slightly change Elisp parser to treat lines that start
with any other printable character but '(' as a start of comment and to
simply ignore the line, just as it treats ';' as a comment. Code blocks
would still be parsed as they are now, and ';' would still mean a comment,
wherever it is encountered, it is just that anything that does not
belong in a code-block (lists) is a comment. For example consider this mail,
if this would be thrown into parser all lines to this point would be simply
ignored since they don't start with '(' or are white spaces.

(my-elisp-fun
        (progn
                (while (very-cool)
                       (do-something-very-literate-here))))

Then I could have Elisp code in between and continue to write this mail
and later on just use this as a source code. Wouldn't that be a step
toward true and more cool literate programming language? Below is another
snippet of imaginary Elisp. If we could do this, then this email would be
a working literate Elisp, where those two snippets are code and text of
this mail is just ignored.

(my-other-fun
        (progn
                ; this-is-some-other-fun
                (to-hopefully-demonstrate-the-idea)))

What would this achieve

More then highly increased coolness factor, it would be a small quality
of life improvement. For example this would actually make it slightly
easier to use org-mode for Elisp programming. For example for us that
use org-mode to structure Emacs init file, we could just throw in our org
file directly, instead of using babel to entangle it into Elisp file first.
If every printable character but '(' starts a comment line, then everything
in org-file but Elisp code would be simply ignored, and only Elisp executed.

If we think other way around, it would also let us use pure Elisp for literate
programming without org-mode whatsoever, albeit it would be a cool feature to
use org-headings and similar to structure the code. It might make code more
structured and thus more readable. When I think in terms of Elisp as a starting
point rather then in terms of org-mode as a starting point, that could result in
adding org-mode organizational features directly to Elisp. One could even mark
say not-implemented functions as todo items, use calendar etc.

We could also entangle other languages within pure Elisp code without using org
mode whatsoever. Either within some code markers for processing them out to
separate files, or without code markers just as documentation or whatever. I
don't have some better example of use-case at the moment.

I don't mean that it is incredibly slow to entangle files, but it would be
slightly more efficient to process Elisp entangled in org mode. I also don't
think it is hard to type ';' at the beginning of a line to start a comment line.
But it is a small convenience and thus quality of life improvement that probably
does not need much changes to a parser but has quite a dramatic effect on how
source code looks in human eye (at least mine, if you don't mind that I count
myself as a part of the species :-)). It would let us use org-mode as a standard
Elisp source code format, which might be just a perceived convenience rather
then some real extra useful thing that does not exist yet.

Some thoughts about implementation

I think that in terms of cost effectiveness with implementation in mind, it
probably isn't that much work to implement this, but honestly I have no idea.
I believe it can't be much work, but I am not sure so I should really put an
exclamation mark to word probably in paragraph above. Feel free to educate me
about cost of making it work. I was looking myself in C source to see if I could
test this myself before I post here, but I couldn't find where you have implemented
parser. I am sorry, but I am that bad :-(.

Essentially when parsing literate Elisp, if I may call it so, what
parser has to do is to simply not flag random printable characters, on lines
that does not belong to code-blocks, as errors in source code. Instead just treat
them as if it has seen a ';'.

It means there are just two classes of printable characters: '(' that opens
a code block, and everything else that opens a comment block. Well almost.
Parsing code blocks would not need to be changed at all, and ';' in code blocks
would still mean that rest of line is a comment, and all code with comments
would still continue to work as it does now. It would only affect new code that
is written in this style. However new Elisp code wouldn't be backward
compatible with old versions of Emacs.

As extra, one could keep current Elisp parser and make new one and use as
in Haskell, en extra 'l' in suffix to denote a literate program, '.lel'. Though
it kind-a looks fun, I don't think it wouldn't be needed, I don't think this
change would need different parser, to ensure backward compatibility. I don't
think that is very important since if we would write Elisp that needs to run on
older versions, we can just let be to write it in literal form.

Drawbacks

As I can think of, it would maybe make spotting errors slightly harder, for
example I could type a random character before an opening parenthesis and
comment out entire line, but those kind of errors are easily spotted on first code
run. Another drawback would be probably syntax highlighting. It could probably
become much harder to detect comments in code since there is no ';' to mark a
comment-line. Maybe I am wrong about this one, it is just a fast thought.

Final thought

I have no idea if somebody else has already thought about this and found that it
can't work. It seems like a very straight-forward and simple thing so I am
probably not the first one to think the thought, and there is probably some
reason why it is not done that I am not aware of. In that case just ignore this
email. It was just a thought I got yesterday. There might be something I am not
aware of that makes this impossible, I am not that familiar with Emacs source to
try to implement this myself unfortunately. It is just an idea, a thought for
discussion, but it would be cool if it can work.

I am sorry for very long email, I hope you have at least somewhat enjoyed my
rather ramblings and Christmas wishes, and please ask Santa to excuse my
English, I am not native English speaker, it is just my 3rd language if it is in
any defense or my horrible writing.

Reply | Threaded
Open this post in threaded view
|

Re: Christmas wish: Literate Elisp

Stefan Monnier
> My proposal is to slightly change Elisp parser to treat lines that start
> with any other printable character but '(' as a start of comment and to
> simply ignore the line, just as it treats ';' as a comment.

The `sexpresso` Haskell package follows the same idea ;-)

As for using it in Elisp: I don't think there's anything stopping anyone
from making such a `literate-elisp-mode` and even arrange for `load` to
handle such a file (just like there is already a package that lets
`load` work directly on .org files).

I'd welcome such a package in GNU ELPA.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: Christmas wish: Literate Elisp

Richard Stallman
In reply to this post by arthur miller
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The idea sounds basically plausible, but note that simply rejecting
all lines that start with anything other than ( plus whitespace
will give the wrong results.  Lisp code includes lines that start
with other things.  Look at any file of Lisp code and you will see
many that start with letters, digits or quotes.  The proposal
needs to be corrected for this.

But I think that correction won't be difficult.

Indented comments should be recognized and formatted differently
but not exactly the same as interpolated text.

Perhaps triple-semicolon should be the marker for interpolated text.
That way, it would not require any change in Lisp parsing or in Lisp Mode.

--
Dr Richard Stallman
Founder, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



Reply | Threaded
Open this post in threaded view
|

Sv: Christmas wish: Literate Elisp

arthur miller
In reply to this post by Stefan Monnier
To be honest my contact with Haskell broke once the university course, some 20 yrs ago was over, so I am a little bit illiterate about Haskells sexy packages. I just recalled that it was possible to invert text and code in Haskell. But cool. Didn't know there is a package to let load org files directly either. What I do in init is use org-babel to entangle my init file (org-babel-load-file (expand-file-name "~/.emacs.d/lisp/init.org")). What is the name of the package? I have tried to google it and looked into elpa packages with list-package but I don't see it. I am just curious how is it done, as I was thinking or they refactor the code as in org-babel into corresponding lisp file.

Thanks for the answers.

Från: Stefan Monnier <[hidden email]>
Skickat: den 12 december 2019 18:29
Till: arthur miller <[hidden email]>
Kopia: emacs-devel <[hidden email]>; [hidden email] <[hidden email]>
Ämne: Re: Christmas wish: Literate Elisp
 
> My proposal is to slightly change Elisp parser to treat lines that start
> with any other printable character but '(' as a start of comment and to
> simply ignore the line, just as it treats ';' as a comment.

The `sexpresso` Haskell package follows the same idea ;-)

As for using it in Elisp: I don't think there's anything stopping anyone
from making such a `literate-elisp-mode` and even arrange for `load` to
handle such a file (just like there is already a package that lets
`load` work directly on .org files).

I'd welcome such a package in GNU ELPA.


        Stefan

Reply | Threaded
Open this post in threaded view
|

Sv: Christmas wish: Literate Elisp

arthur miller
In reply to this post by Richard Stallman
"The idea sounds basically plausible, but note that simply rejecting
all lines that start with anything other than ( plus whitespace
will give the wrong results.  Lisp code includes lines that start
with other things.  Look at any file of Lisp code and you will see
many that start with letters, digits or quotes.  The proposal
needs to be corrected for this."

To be honest I am not sure if I am a bit not understood here, or just
me not being knowledgable enough about lisp (could be both 🙂). I
will try to clarify.

What I propose is to treat lines that does not belong to a code block ().
I don't mean every line that does not start with '('. there can be many
lines in a code block between an '(' and final matching ')'. Once '(' is
encountered, parsing should be just as it is now, without difference.
';' would still be the comment indicator, so no change needed there.
Just for the text that does not belong to any matching pair of (), so to
say in-between "block final"  ')' and "block opening"  '('. I am sorry if
my terminology is not very lisp-like, I am used to think more in terms
of a C compiler.

I am not super-versatile with elisp to be honest, so I will have to
ask: is it possible to have elisp that isn't in-between '(' and ')'.
Is it possible to have symbols or literals in elisp outside ()? I
am sorry if I am total noob, but I am not used to see those in
sources, is it just rare to have them, or do I missunderand? I am
looking around in some elisp but I can't find any.


Från: Richard Stallman <[hidden email]>
Skickat: den 14 december 2019 05:16
Till: arthur miller <[hidden email]>
Kopia: [hidden email] <[hidden email]>
Ämne: Re: Christmas wish: Literate Elisp
 
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The idea sounds basically plausible, but note that simply rejecting
all lines that start with anything other than ( plus whitespace
will give the wrong results.  Lisp code includes lines that start
with other things.  Look at any file of Lisp code and you will see
many that start with letters, digits or quotes.  The proposal
needs to be corrected for this.

But I think that correction won't be difficult.

Indented comments should be recognized and formatted differently
but not exactly the same as interpolated text.

Perhaps triple-semicolon should be the marker for interpolated text.
That way, it would not require any change in Lisp parsing or in Lisp Mode.

--
Dr Richard Stallman
Founder, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)


Reply | Threaded
Open this post in threaded view
|

Re: Sv: Christmas wish: Literate Elisp

Stefan Monnier
In reply to this post by arthur miller
arthur miller [2019-12-14 04:40:30] wrote:

> To be honest my contact with Haskell broke once the university course, some
> 20 yrs ago was over, so I am a little bit illiterate about Haskells sexy
> packages.  I just recalled that it was possible to invert text and code in
> Haskell.  But cool.

The fact that it's written is Haskell is just incidental.

> Didn't know there is a package to let load org files directly either.

Hmm... The closest I can find is https://github.com/jingtaozf/literate-elisp/
but it doesn't quite match what I think I was referring to.


        Stefan


> Thanks for the answers.
> ________________________________
> Från: Stefan Monnier <[hidden email]>
> Skickat: den 12 december 2019 18:29
> Till: arthur miller <[hidden email]>
> Kopia: emacs-devel <[hidden email]>; [hidden email] <[hidden email]>
> Ämne: Re: Christmas wish: Literate Elisp
>
>> My proposal is to slightly change Elisp parser to treat lines that start
>> with any other printable character but '(' as a start of comment and to
>> simply ignore the line, just as it treats ';' as a comment.
>
> The `sexpresso` Haskell package follows the same idea ;-)
>
> As for using it in Elisp: I don't think there's anything stopping anyone
> from making such a `literate-elisp-mode` and even arrange for `load` to
> handle such a file (just like there is already a package that lets
> `load` work directly on .org files).
>
> I'd welcome such a package in GNU ELPA.
>
>
>         Stefan


Reply | Threaded
Open this post in threaded view
|

RE: Sv: Christmas wish: Literate Elisp

arthur miller
Thanks, I have also found literste-li when I searched, but thought it didn't looked  as suggested .

I have played a bit yesterday,  for about an hour and managed to find readevalloop in lread.c in sources. I managed to get a partial hack to work in some cases, but in other I get parse errors. I am not sure I understand exactly how the parser hangs together yet, if I can do everything in that function or I have to change some of other functions too. I will have more time tonight evening.



Skickat från min Samsung Galaxy-smartphone.



-------- Originalmeddelande --------
Från: Stefan Monnier <[hidden email]>
Datum: 2019-12-14 15:08 (GMT+01:00)
Till: arthur miller <[hidden email]>
Kopia: emacs-devel <[hidden email]>, [hidden email]
Ämne: Re: Sv: Christmas wish: Literate Elisp

arthur miller [2019-12-14 04:40:30] wrote:

> To be honest my contact with Haskell broke once the university course, some
> 20 yrs ago was over, so I am a little bit illiterate about Haskells sexy
> packages.  I just recalled that it was possible to invert text and code in
> Haskell.  But cool.

The fact that it's written is Haskell is just incidental.

> Didn't know there is a package to let load org files directly either.

Hmm... The closest I can find is https://github.com/jingtaozf/literate-elisp/
but it doesn't quite match what I think I was referring to.


        Stefan


> Thanks for the answers.
> ________________________________
> Från: Stefan Monnier <[hidden email]>
> Skickat: den 12 december 2019 18:29
> Till: arthur miller <[hidden email]>
> Kopia: emacs-devel <[hidden email]>; [hidden email] <[hidden email]>
> Ämne: Re: Christmas wish: Literate Elisp
>
>> My proposal is to slightly change Elisp parser to treat lines that start
>> with any other printable character but '(' as a start of comment and to
>> simply ignore the line, just as it treats ';' as a comment.
>
> The `sexpresso` Haskell package follows the same idea ;-)
>
> As for using it in Elisp: I don't think there's anything stopping anyone
> from making such a `literate-elisp-mode` and even arrange for `load` to
> handle such a file (just like there is already a package that lets
> `load` work directly on .org files).
>
> I'd welcome such a package in GNU ELPA.
>
>
>         Stefan

Reply | Threaded
Open this post in threaded view
|

Sv: Sv: Christmas wish: Literate Elisp

arthur miller
In reply to this post by Stefan Monnier
Hello again,

I was able to make readevalloop do what I want. It was rather trivial to
implement this (once I realized how it works 🙂). Emacs built fine and I was
able to test wtih eval-buffer and eval-regionwhich worked as intended.

It was just 4 code lines, I just copied codeto parse comment and change the
condition in if-statement:

(message "Hello, World!")

(message "Hello Again!")

In code blocs the ';' is still a comment delimiter:

(message
 ;; Here is a line comment
 "I am a bit chatty today!")

That's it for today folks!

(message "Bye bye cruel world!")

This email was composed in scratch buffer and executed during writing with
eval-buffer for testing.

However, byte compiler is not happy about this. It emits warnings
for every word out of code blocks as references to free variables. I am not sure
where to look to patch it, maybe another day.

Från: Stefan Monnier <[hidden email]>
Skickat: den 14 december 2019 15:08
Till: arthur miller <[hidden email]>
Kopia: emacs-devel <[hidden email]>; [hidden email] <[hidden email]>
Ämne: Re: Sv: Christmas wish: Literate Elisp
 
arthur miller [2019-12-14 04:40:30] wrote:

> To be honest my contact with Haskell broke once the university course, some
> 20 yrs ago was over, so I am a little bit illiterate about Haskells sexy
> packages.  I just recalled that it was possible to invert text and code in
> Haskell.  But cool.

The fact that it's written is Haskell is just incidental.

> Didn't know there is a package to let load org files directly either.

Hmm... The closest I can find is https://github.com/jingtaozf/literate-elisp/
but it doesn't quite match what I think I was referring to.


        Stefan


> Thanks for the answers.
> ________________________________
> Från: Stefan Monnier <[hidden email]>
> Skickat: den 12 december 2019 18:29
> Till: arthur miller <[hidden email]>
> Kopia: emacs-devel <[hidden email]>; [hidden email] <[hidden email]>
> Ämne: Re: Christmas wish: Literate Elisp
>
>> My proposal is to slightly change Elisp parser to treat lines that start
>> with any other printable character but '(' as a start of comment and to
>> simply ignore the line, just as it treats ';' as a comment.
>
> The `sexpresso` Haskell package follows the same idea ;-)
>
> As for using it in Elisp: I don't think there's anything stopping anyone
> from making such a `literate-elisp-mode` and even arrange for `load` to
> handle such a file (just like there is already a package that lets
> `load` work directly on .org files).
>
> I'd welcome such a package in GNU ELPA.
>
>
>         Stefan


lread.patch (534 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Sv: Sv: Christmas wish: Literate Elisp

Stefan Monnier
>> As for using it in Elisp: I don't think there's anything stopping anyone
>> from making such a `literate-elisp-mode` and even arrange for `load` to
>> handle such a file (just like there is already a package that lets
>> `load` work directly on .org files).
> I was able to make readevalloop do what I want. It was rather trivial to

Note that in my comment above I meant it in existing Emacsen,
i.e. without any modification to the C code.
The idea is to make use of existing hooks such as
`load-source-file-function` or `file-name-handler-alist`.

I'm not sure what hook could be used to do the same with
byte-compilation, but in the worst case, an advice should do the trick.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Sv: Sv: Sv: Christmas wish: Literate Elisp

arthur miller

Aha 😊

Sorry, I didn’t realized what you ment. I am not sure I know that much of elisp,

but I may try to look at, would be lots of learning for me 😊.

 

Skickades från E-post för Windows 10

 

Från: [hidden email]
Skickat: den 16 december 2019 14:41
Till: [hidden email]
Kopia: [hidden email]; [hidden email]
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

 

>> As for using it in Elisp: I don't think there's anything stopping anyone
>> from making such a `literate-elisp-mode` and even arrange for `load` to
>> handle such a file (just like there is already a package that lets
>> `load` work directly on .org files).
> I was able to make readevalloop do what I want. It was rather trivial to

Note that in my comment above I meant it in existing Emacsen,
i.e. without any modification to the C code.
The idea is to make use of existing hooks such as
`load-source-file-function` or `file-name-handler-alist`.

I'm not sure what hook could be used to do the same with
byte-compilation, but in the worst case, an advice should do the trick.


        Stefan

 

Reply | Threaded
Open this post in threaded view
|

Re: Sv: Sv: Christmas wish: Literate Elisp

Jonathan Leech-Pepin
Arthur,

Have you happened to have looked at Outshine (https://github.com/alphapapa/outshine)?

It enhances outline-minor-mode and lets it work with standard org-syntax while in comment blocks (and defines hotkeys to swap into an edit buffer that makes it look like a normal Org-Mode buffer).

By keeping the literate syntax in comments you don't break the existing parser.

Regards,
Jon

On Mon, Dec 16, 2019 at 9:03 AM arthur miller <[hidden email]> wrote:

Aha 😊

Sorry, I didn’t realized what you ment. I am not sure I know that much of elisp,

but I may try to look at, would be lots of learning for me 😊.

 

Skickades från E-post för Windows 10

 

Från: [hidden email]
Skickat: den 16 december 2019 14:41
Till: [hidden email]
Kopia: [hidden email]; [hidden email]
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

 

>> As for using it in Elisp: I don't think there's anything stopping anyone
>> from making such a `literate-elisp-mode` and even arrange for `load` to
>> handle such a file (just like there is already a package that lets
>> `load` work directly on .org files).
> I was able to make readevalloop do what I want. It was rather trivial to

Note that in my comment above I meant it in existing Emacsen,
i.e. without any modification to the C code.
The idea is to make use of existing hooks such as
`load-source-file-function` or `file-name-handler-alist`.

I'm not sure what hook could be used to do the same with
byte-compilation, but in the worst case, an advice should do the trick.


        Stefan

 

Reply | Threaded
Open this post in threaded view
|

RE: Sv: Sv: Christmas wish: Literate Elisp

arthur miller
Of course, I am aware of outline modes, but my proposal is more about literate programming then outlining. Being able to use org outlining in elisp is just an extra icing on the cake that comes out implicitly as a bi product for free. Otherwise of course one can use outline code for folding. 

It is true, that keeping directives in comments does not break the existing parser, but then it is not so much of literate programming, is it?

If we would just use directives in comments then we already have org + babel. But I think 4 extra lines of C code to enable this feature in eval-loop was a trivial price to pay :-). 

Now I just wonder how to change the bytecode compiler. Unfortunately it seems to be much more involved process. I don't know  if I can do it, since I am  not so acquainted with Emacs internals, so it would be cool if someone more knowledgeable can at least hint me if not help.


Skickat från min Samsung Galaxy-smartphone.



-------- Originalmeddelande --------
Från: Jonathan Leech-Pepin <[hidden email]>
Datum: 2019-12-16 17:07 (GMT+01:00)
Till: arthur miller <[hidden email]>
Kopia: Stefan Monnier <[hidden email]>, [hidden email], emacs-devel <[hidden email]>
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

Arthur,

Have you happened to have looked at Outshine (https://github.com/alphapapa/outshine)?

It enhances outline-minor-mode and lets it work with standard org-syntax while in comment blocks (and defines hotkeys to swap into an edit buffer that makes it look like a normal Org-Mode buffer).

By keeping the literate syntax in comments you don't break the existing parser.

Regards,
Jon

On Mon, Dec 16, 2019 at 9:03 AM arthur miller <[hidden email]> wrote:

Aha 😊

Sorry, I didn’t realized what you ment. I am not sure I know that much of elisp,

but I may try to look at, would be lots of learning for me 😊.

 

Skickades från E-post för Windows 10

 

Från: [hidden email]
Skickat: den 16 december 2019 14:41
Till: [hidden email]
Kopia: [hidden email]; [hidden email]
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

 

>> As for using it in Elisp: I don't think there's anything stopping anyone
>> from making such a `literate-elisp-mode` and even arrange for `load` to
>> handle such a file (just like there is already a package that lets
>> `load` work directly on .org files).
> I was able to make readevalloop do what I want. It was rather trivial to

Note that in my comment above I meant it in existing Emacsen,
i.e. without any modification to the C code.
The idea is to make use of existing hooks such as
`load-source-file-function` or `file-name-handler-alist`.

I'm not sure what hook could be used to do the same with
byte-compilation, but in the worst case, an advice should do the trick.


        Stefan

 

Reply | Threaded
Open this post in threaded view
|

Re: Sv: Sv: Christmas wish: Literate Elisp

Adam Porter
arthur miller <[hidden email]> writes:

> It is true, that keeping directives in comments does not break the
> existing parser, but then it is not so much of literate programming,
> is it?

Literate programming is not a matter of syntax.  For example, this
summary in the Wikipedia article seems to describe it well:

  Literate programming is a programming paradigm introduced by Donald
  Knuth in which a computer program is given an explanation of its logic
  in a natural language, such as English, interspersed with snippets of
  macros and traditional source code, from which compilable source code
  can be generated.

  The literate programming paradigm, as conceived by Knuth, represents a
  move away from writing computer programs in the manner and order
  imposed by the computer, and instead enables programmers to develop
  programs in the order demanded by the logic and flow of their
  thoughts. Literate programs are written as an uninterrupted exposition
  of logic in an ordinary human language, much like the text of an
  essay, in which macros are included to hide abstractions and
  traditional source code.

  Literate programming (LP) tools are used to obtain two representations
  from a literate source file: one suitable for further compilation or
  execution by a computer, the "tangled" code, and another for viewing
  as formatted documentation, which is said to be "woven" from the
  literate source. While the first generation of literate programming
  tools were computer language-specific, the later ones are
  language-agnostic and exist above the programming languages.

None of that requires un-prefixed comment syntax.  Consider examples of
actual, full-scale literate programming projects, which are written in a
variety of languages and source code formats.

> If we would just use directives in comments then we already have org +
> babel. But I think 4 extra lines of C code to enable this feature in
> eval-loop was a trivial price to pay :-).

Changing the canonical parser of a widely used language would have
effects reaching far beyond the parser's native software.  Consider all
other software which parses Elisp, e.g. syntax highlighting tools
outside of Emacs, which would not correctly highlight these un-prefixed
comments you propose.  As well, consider other implementations, like
Guile's, which would also have to be adjusted.  Then consider all the
code linting tools which would likely need fixing.  It's not merely a
matter of 4 lines of code in an Emacs source file.

As has been mentioned, benefits such as outlining are easily achieved
with existing tools, some of which are even built-in to Emacs
(e.g. outline-minor-mode for simple outlining in Elisp files, and
org-mode for prose-first, noweb-style source files).

Are you really so concerned about omitting a semicolon at the beginning
of top-level comment lines that you want to change the Elisp parser?


Reply | Threaded
Open this post in threaded view
|

Sv: Sv: Sv: Christmas wish: Literate Elisp

arthur miller

Hello, sorry for a bit late answer, but I was very busy Monday/Tuesday.

 

To make this more readable I will try to summerize here as one-piece text

for easier reading.

 

As you mention yourself from,Wikipedia, literate programming is about moving

away from programming for computers to programming for humans. Using comments

to comment-away text ment for humans or to mark lines between comments and

code is indeed writing for the computer, not the human. Humans tend to be

distracted by unnecessary clutter, and managing stuff just for the sake of

machine does take a little more effort then not doing it.

 

Being programming-language agnostic though is probably not possible

without having some kind of special tool, or special syntax due to some

languages being probably very hard to distinguish from ordinary text. For

example I believe that C/C++ or even Bash or Makefiles are very hard if

not impossible to process that way without some very complicated parser. I

can also imagine case where Elisp is impossible to distinguish from text,

(for example this line is valid text but would fail in parser), or whatever

we would put into a parenthesis on a separate line that does not belong to code.

 

Anyway, due to elisp looks, we have those "natural markers", ( and ) which

we can use as delimiters between code and text.

 

For the question of being concerned about omitting ';' to change the parser.

Well yes :-). Small syntactic sugar change but it reduces that visual clutter.

It is one step in that direction of writing code for humans and not for the

computer. See it as experiment. I thought it would be a cool thing to have. It

is just an idea. The change is not in an intrusive way. All your elisp is

still very same. It just adds an extra feature pretty much for free. If it would

be an intrusive change that requires your old elisp to change, than I wouldn't

suggest it. By the way, one could also add a variable to turn off that feature.

 

As for the tools, since old elisp code does not change, all your tools would

still continue to work on all your ordinary elisp, and you could still use ';'

just as you do now. What would change is that you would be able to write some

elisp as you can't now, and only on that code the tools would probably get

confused. I have no idea how hard would it be to change font-locking and

indentation checkers, maybe hard, maybe not at all. I can't answer that since I

am not acquianted with how those tools are implemented (lots of regex as I have

heard). But in worst case, you can always fall back on writing ordinary elisp.

 

As a note, one will still have to use ';' in directives, like ;;;###autoload

or ;;; -*- lexical-binding: t; -*- . Otherwise it would need lots more work
to get it to work, which I am not sure is worth.

 

Från: [hidden email]
Skickat: den 17 december 2019 12:07
Till: [hidden email]
Ämne: Re: Sv: Sv: Christmas wish: Literate Elisp

 

arthur miller <[hidden email]> writes:

> It is true, that keeping directives in comments does not break the
> existing parser, but then it is not so much of literate programming,
> is it?

Literate programming is not a matter of syntax.  For example, this
summary in the Wikipedia article seems to describe it well:

  Literate programming is a programming paradigm introduced by Donald
  Knuth in which a computer program is given an explanation of its logic
  in a natural language, such as English, interspersed with snippets of
  macros and traditional source code, from which compilable source code
  can be generated.

  The literate programming paradigm, as conceived by Knuth, represents a
  move away from writing computer programs in the manner and order
  imposed by the computer, and instead enables programmers to develop
  programs in the order demanded by the logic and flow of their
  thoughts. Literate programs are written as an uninterrupted exposition
  of logic in an ordinary human language, much like the text of an
  essay, in which macros are included to hide abstractions and
  traditional source code.

  Literate programming (LP) tools are used to obtain two representations
  from a literate source file: one suitable for further compilation or
  execution by a computer, the "tangled" code, and another for viewing
  as formatted documentation, which is said to be "woven" from the
  literate source. While the first generation of literate programming
  tools were computer language-specific, the later ones are
  language-agnostic and exist above the programming languages.

None of that requires un-prefixed comment syntax.  Consider examples of
actual, full-scale literate programming projects, which are written in a
variety of languages and source code formats.

> If we would just use directives in comments then we already have org +
> babel. But I think 4 extra lines of C code to enable this feature in
> eval-loop was a trivial price to pay :-).

Changing the canonical parser of a widely used language would have
effects reaching far beyond the parser's native software.  Consider all
other software which parses Elisp, e.g. syntax highlighting tools
outside of Emacs, which would not correctly highlight these un-prefixed
comments you propose.  As well, consider other implementations, like
Guile's, which would also have to be adjusted.  Then consider all the
code linting tools which would likely need fixing.  It's not merely a
matter of 4 lines of code in an Emacs source file.

As has been mentioned, benefits such as outlining are easily achieved
with existing tools, some of which are even built-in to Emacs
(e.g. outline-minor-mode for simple outlining in Elisp files, and
org-mode for prose-first, noweb-style source files).

Are you really so concerned about omitting a semicolon at the beginning
of top-level comment lines that you want to change the Elisp parser?

 

Reply | Threaded
Open this post in threaded view
|

Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

Adam Porter
arthur miller <[hidden email]> writes:

> As you mention yourself from,Wikipedia, literate programming is about moving
> away from programming for computers to programming for humans. Using comments
> to comment-away text ment for humans or to mark lines between comments and
> code is indeed writing for the computer, not the human. Humans tend to be
> distracted by unnecessary clutter, and managing stuff just for the sake of
> machine does take a little more effort then not doing it.

; I am not more distracted while reading this line of text
than I am while reading this one.

On the contrary, the semicolon prefix (and fontification according to my
preferences in Elisp buffers) makes it easy for me to see what is code
and what is prose.

> I can also imagine case where Elisp is impossible to distinguish from text,
> (for example this line is valid text but would fail in parser), or whatever
> we would put into a parenthesis on a separate line that does not
> belong to code.

That line beginning with a paren parses as valid Elisp.  Whether it
evaluates depends on the context.

> Anyway, due to elisp looks, we have those "natural markers", ( and ) which
> we can use as delimiters between code and text.

This is the problem: as you have shown, parentheses commonly appear in
prose, even at the beginning of lines.  Punctuation, such as semicolons,
do not
; or, at least, should not.

> For the question of being concerned about omitting ';' to change the parser.
> Well yes :-). Small syntactic sugar change but it reduces that visual
> clutter.

You call it clutter; I call it clearly delineating code and prose.  What
would look cluttered to me would be prose (which may include commented-out
code) interspersed with
(insert "code, which may include or generate prose.")

> It is one step in that direction of writing code for humans and not for the
> computer.

Until we can write programs in human language, we must write both.  For
humans' sake, the two need to be clearly separated.  If you don't like
comment prefix syntax, you can use Org files and their source blocks, so
your prose and your code can both start at column 0.  You can even
choose a face for org-meta-line that makes #+BEGIN_SRC lines virtually
(or literally!) disappear.

> See it as experiment. I thought it would be a cool thing to have. It
> is just an idea. The change is not in an intrusive way. All your elisp
> is still very same.  It just adds an extra feature pretty much for
> free. If it would be an intrusive change that requires your old elisp
> to change, than I wouldn't suggest it. By the way, one could also add
> a variable to turn off that feature.

No, in fact, some of my Elisp would have to change, because it would no
longer be sufficient to look at comment prefixes to know whether a line
is code or comment.  Such a change would be very expensive in terms of
the necessary human work that would ripple outward.  And your proposed
variable would not obviate the need to account for both states in my
code.

And since parens can appear at the start of lines in prose, how would
anyone but a human know whether a line is code or prose?

> As for the tools, since old elisp code does not change, all your tools
> would still continue to work on all your ordinary elisp, and you could
> still use ';' just as you do now. What would change is that you would
> be able to write some elisp as you can't now, and only on that code
> the tools would probably get confused. I have no idea how hard would
> it be to change font-locking and indentation checkers, maybe hard,
> maybe not at all. I can't answer that since I am not acquianted with
> how those tools are implemented (lots of regex as I have heard).

> But in worst case, you can always fall back on writing ordinary elisp.

My tools which handle Elisp would have no such luxury, because they must
handle all valid Elisp.

> I have no idea how hard would it be to change font-locking and
> indentation checkers, maybe hard, maybe not at all.

Please, if you won't take my word for it, try it and see.


Reply | Threaded
Open this post in threaded view
|

Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

arthur miller

> On the contrary, the semicolon prefix (and fontification according to my
> preferences in Elisp buffers) makes it easy for me to see what is code
> and what is prose.

Aren’t starting ’(’ and ’)’ + identation enough?


> No, in fact, some of my Elisp would have to change, because it would no
> onger be sufficient to look at comment prefixes to know whether a line
> s code or comment.  Such a change would be very expensive in terms of
> the necessary human work that would ripple outward.  And your proposed
> variable would not obviate the need to account for both states in my
> code.

In which way? Can you alaborate more please? You would not need to write

anything special. If you don’t wish to write literal code, then just don’t. Prefix

your comment lines with ’;’ as currently and don’t care about it, Everything will

work as it does now. I don’t see a problem there. If I am wrong please explain.

> This is the problem: as you have shown, parentheses commonly appear in
> prose, even at the beginning of lines.  Punctuation, such as semicolons,
> do not
> ; or, at least, should not

 

Yes, and it is always possible to construct some obscure case where things break.

Look for example at C or C++ and all undefined behaviours in the standard. I mean,

sure, there are always soem obscurities, but do they matter? If one writes an

article or a book or some prosa and uses literal programming, and there is one

explicit case not allowed to use, how important is than to use exactly that

constrcut? I mean considering cons and pros, should one abandon the entire

idea because of one minor obscurity?

> Until we can write programs in human language, we must write both.  For
> humans' sake, the two need to be clearly separated.  If you don't like
> comment prefix syntax, you can use Org files and their source blocks, so
> your prose and your code can both start at column 0.  You can even
> choose a face for org-meta-line that makes #+BEGIN_SRC lines virtually
> (or literally!) disappear.

 

Well as I see literal programming it is to make it easier for humans to mix code and

prosa. What I proposedis just one small step further in that direction. I don’t see how

org makes it any different? In org you still have to use both ’;’ and #+BEGIN_SRC  -

#+END_SRC markers. So it is even more clutter to eye. It is not about starting at column 0,

It is about being able to simple write and mix code and text. If it is good or bad idea is up

to individual preference. I personally find it a cool thing, you seem to think it violates old

ways, so obviously we wont agree on that one. But thank you for the discussion, I surely

appreciate the input, even if I don’t agree on the main conclusion. I gladly read more!

 

As a small reflection about us humans being slentrians, I have an illustration.
Caves are still as good for living as they were 20 000 years ago. We can for sure

make many cases against living in houses such as not being natural, can crash on it’s

inhabitants, cost resources to construct etc etc. But yet, pros of liiving in houses outweigh

far the cons, and not many people today prefer to live in a cave. Sorry maybe ti is a too

contrived illustration. Anyway, if you are good with ’;’ just continue to use it. But if a

change like this does not seem to cost lots in development terms, then why opposing if

somebody else find it cool. It is just that is more elegant and in a sense cool thing to be

able to do it without special tools as some markers.

 

Från: [hidden email]
Skickat: den 18 december 2019 19:50
Till: [hidden email]
Ämne: Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

 

arthur miller <[hidden email]> writes:

> As you mention yourself from,Wikipedia, literate programming is about moving
> away from programming for computers to programming for humans. Using comments
> to comment-away text ment for humans or to mark lines between comments and
> code is indeed writing for the computer, not the human. Humans tend to be
> distracted by unnecessary clutter, and managing stuff just for the sake of
> machine does take a little more effort then not doing it.

; I am not more distracted while reading this line of text
than I am while reading this one.

On the contrary, the semicolon prefix (and fontification according to my
preferences in Elisp buffers) makes it easy for me to see what is code
and what is prose.

> I can also imagine case where Elisp is impossible to distinguish from text,
> (for example this line is valid text but would fail in parser), or whatever
> we would put into a parenthesis on a separate line that does not
> belong to code.

That line beginning with a paren parses as valid Elisp.  Whether it
evaluates depends on the context.

> Anyway, due to elisp looks, we have those "natural markers", ( and ) which
> we can use as delimiters between code and text.

This is the problem: as you have shown, parentheses commonly appear in
prose, even at the beginning of lines.  Punctuation, such as semicolons,
do not
; or, at least, should not.

> For the question of being concerned about omitting ';' to change the parser.
> Well yes :-). Small syntactic sugar change but it reduces that visual
> clutter.

You call it clutter; I call it clearly delineating code and prose.  What
would look cluttered to me would be prose (which may include commented-out
code) interspersed with
(insert "code, which may include or generate prose.")

> It is one step in that direction of writing code for humans and not for the
> computer.

Until we can write programs in human language, we must write both.  For
humans' sake, the two need to be clearly separated.  If you don't like
comment prefix syntax, you can use Org files and their source blocks, so
your prose and your code can both start at column 0.  You can even
choose a face for org-meta-line that makes #+BEGIN_SRC lines virtually
(or literally!) disappear.

> See it as experiment. I thought it would be a cool thing to have. It
> is just an idea. The change is not in an intrusive way. All your elisp
> is still very same.  It just adds an extra feature pretty much for
> free. If it would be an intrusive change that requires your old elisp
> to change, than I wouldn't suggest it. By the way, one could also add
> a variable to turn off that feature.

No, in fact, some of my Elisp would have to change, because it would no
longer be sufficient to look at comment prefixes to know whether a line
is code or comment.  Such a change would be very expensive in terms of
the necessary human work that would ripple outward.  And your proposed
variable would not obviate the need to account for both states in my
code.

And since parens can appear at the start of lines in prose, how would
anyone but a human know whether a line is code or prose?

> As for the tools, since old elisp code does not change, all your tools
> would still continue to work on all your ordinary elisp, and you could
> still use ';' just as you do now. What would change is that you would
> be able to write some elisp as you can't now, and only on that code
> the tools would probably get confused. I have no idea how hard would
> it be to change font-locking and indentation checkers, maybe hard,
> maybe not at all. I can't answer that since I am not acquianted with
> how those tools are implemented (lots of regex as I have heard).

> But in worst case, you can always fall back on writing ordinary elisp.

My tools which handle Elisp would have no such luxury, because they must
handle all valid Elisp.

> I have no idea how hard would it be to change font-locking and
> indentation checkers, maybe hard, maybe not at all.

Please, if you won't take my word for it, try it and see.

 

Reply | Threaded
Open this post in threaded view
|

Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

Stefan Monnier
In reply to this post by Adam Porter
> ; I am not more distracted while reading this line of text
> than I am while reading this one.

Thre is no discussion of changing the existing syntax/semantic of
Elisp here.  Arthur is proposing a new file format/syntax as an
alternative to the use of weave/tangle or Org-mode.

You don't have to agree with each other.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Christmas wish: Literate Elisp (Intro)

VanL

>> ; I am not more distracted while reading this line of text
>> than I am while reading this one.
>
> Thre is no discussion of changing the existing syntax/semantic of
> Elisp here.  Arthur is proposing a new file format/syntax as an
> alternative to the use of weave/tangle or Org-mode.
>
> You don't have to agree with each other.

Arthur's vision is a beautiful thing to my eyes.  Lawyers and
programmers can read and write, thoughtfully and clearly, together
collaboratively, at the Creation before the two crashs.

(message
        "Before you go, have you heard the comedian tell the joke about
         the lawyer, the scientist, and the engineer going to Mars?")

--
VanL., 🐞
  əə0@ 一 二 三 言 語 𝔖 元 示 証 明 海 自 己 漢 本 人 Gnus/Emacs (berkeley-unix)


Reply | Threaded
Open this post in threaded view
|

Re: Sv: Sv: Sv: Christmas wish: Literate Elisp

Adam Porter
In reply to this post by Stefan Monnier
Stefan Monnier <[hidden email]> writes:

> Thre is no discussion of changing the existing syntax/semantic of
> Elisp here.  Arthur is proposing a new file format/syntax as an
> alternative to the use of weave/tangle or Org-mode.

That's not my understanding of what Arthur is proposing.  You did
propose an alternative file format in earlier messages when you wrote:

> As for using it in Elisp: I don't think there's anything stopping
> anyone from making such a `literate-elisp-mode` and even arrange for
> `load` to handle such a file (just like there is already a package
> that lets `load` work directly on .org files).  I'd welcome such a
> package in GNU ELPA.

And:

> Note that in my comment above I meant it in existing Emacsen,
> i.e. without any modification to the C code.  The idea is to make use
> of existing hooks such as `load-source-file-function` or
> `file-name-handler-alist`.  I'm not sure what hook could be used to do
> the same with byte-compilation, but in the worst case, an advice
> should do the trick.

But Arthur has been proposing to alter the Emacs Elisp parser itself,
e.g. when he wrote:

> My proposal is to slightly change Elisp parser to treat lines that
> start with any other printable character but '(' as a start of comment
> and to simply ignore the line, just as it treats ';' as a comment

And:

> I was able to make readevalloop do what I want. It was rather trivial
> to implement this (once I realized how it works 🙂). Emacs built fine
> and I was able to test wtih eval-buffer and eval-regionwhich worked as
> intended.

> It was just 4 code lines, I just copied codeto parse comment and
> change the condition in if-statement:

> However, byte compiler is not happy about this. It emits warnings for
> every word out of code blocks as references to free variables. I am
> not sure where to look to patch it, maybe another day.

And:

> If we would just use directives in comments then we already have org +
> babel. But I think 4 extra lines of C code to enable this feature in
> eval-loop was a trivial price to pay :-).

> Now I just wonder how to change the bytecode compiler. Unfortunately
> it seems to be much more involved process. I don't know if I can do
> it, since I am not so acquainted with Emacs internals, so it would be
> cool if someone more knowledgeable can at least hint me if not help.

I've looked again at all messages in the thread.  Am I missing
something?


Reply | Threaded
Open this post in threaded view
|

Re: Sv: Sv: Sv: Sv: Christmas wish: Literate Elisp

Adam Porter
In reply to this post by arthur miller
arthur miller <[hidden email]> writes:

>> On the contrary, the semicolon prefix (and fontification according to
>> my preferences in Elisp buffers) makes it easy for me to see what is
>> code and what is prose.
>
> Aren’t starting ’(’ and ’)’ + identation enough?

I explained in the message you quoted why they are not enough.  Please
read it carefully.

>> No, in fact, some of my Elisp would have to change, because it would
>> no onger be sufficient to look at comment prefixes to know whether a
>> line s code or comment.  Such a change would be very expensive in
>> terms of the necessary human work that would ripple outward.  And
>> your proposed variable would not obviate the need to account for both
>> states in my code.
>
> In which way? Can you alaborate more please? You would not need to
> write anything special. If you don’t wish to write literal code, then
> just don’t. Prefix your comment lines with ’;’ as currently and don’t
> care about it, Everything will work as it does now. I don’t see a
> problem there. If I am wrong please explain.

You don't seem to understand.  I am talking about existing code outside
of emacs.git which parses Elisp, which would not be compatible with your
proposed changes to Elisp syntax.

>> This is the problem: as you have shown, parentheses commonly appear in
>> prose, even at the beginning of lines.  Punctuation, such as semicolons,
>> do not
>> ; or, at least, should not
>
> Yes, and it is always possible to construct some obscure case where
> things break.  Look for example at C or C++ and all undefined
> behaviours in the standard. I mean, sure, there are always soem
> obscurities, but do they matter? If one writes an article or a book or
> some prosa and uses literal programming, and there is one explicit
> case not allowed to use, how important is than to use exactly that
> constrcut? I mean considering cons and pros, should one abandon the
> entire idea because of one minor obscurity?

Your own message showed how it happens in prose.  In fact, it's common
enough and ambiguous enough that Elisp already forbids open-parens at
column 0 in strings and requires them to be escaped.  How is this a
minor obscurity?

Don't you realize how important it is that syntax in a file format be
unambiguous?

> Well as I see literal programming it is to make it easier for humans
> to mix code and prosa. What I proposedis just one small step further
> in that direction.

It is also a step toward ambiguity, churn, and bugs in other people's
code, which you would not have to fix.

> I don’t see how org makes it any different? In org you still have to
> use both ’;’ and #+BEGIN_SRC - #+END_SRC markers.

Have you used Org mode?  Have you looked at a literate program written
in an Org file?  The point is that top-level comments are written
between code blocks, without comment prefixes.

> So it is even more clutter to eye.

As I said in the message you're replying to:

  You can even choose a face for org-meta-line that makes #+BEGIN_SRC
  lines virtually (or literally!) disappear.

> It is about being able to simple write and mix code and text. If it is
> good or bad idea is up to individual preference. I personally find it
> a cool thing, you seem to think it violates old ways,

I have said nothing about old ways.  I explained exactly what the
problems with the proposal are.  Please read my message carefully.

> As a small reflection about us humans being slentrians, I have an
> illustration.  Caves are still as good for living as they were 20 000
> years ago. We can for sure make many cases against living in houses
> such as not being natural, can crash on it’s inhabitants, cost
> resources to construct etc etc. But yet, pros of liiving in houses
> outweigh far the cons, and not many people today prefer to live in a
> cave. Sorry maybe ti is a too contrived illustration. Anyway, if you
> are good with ’;’ just continue to use it. But if a change like this
> does not seem to cost lots in development terms, then why opposing if
> somebody else find it cool. It is just that is more elegant and in a
> sense cool thing to be able to do it without special tools as some
> markers.

It's not cool to imply that citing technical problems with your proposal
is like being a cave-dweller.


123