I had an idea to improve regexp-opt. (I use Emacs 25.3.2).
In a regexp when you have a group with alternatives, sometimes all alternatives *finish* with one or more common atom regexps. You could take the common part out of the group and try to improve the remaining smaller group, splitting all strings that match and recurse regexp-opt.
Example 1: we read from regexp-opt.el:
;; One possible improvement would be to compile '("aa" "ab" "ba" "bb")
;; into "[ab][ab]" rather than "a[ab]\\|b[ab]". I'm not sure it's worth
;; it but if someone knows how to do it without going through too many
;; contortions, I'm all ears.
(regexp-opt '("aa" "ab" "ba" "bb"))
All alternatives finish with "[ab]", so it is equivalent to
The remaining group is "\\(?:a\\|b\\)". We can further improve making a list of all strings that match it and recourse regexp-opt:
> (regexp-opt '("car" "cdr" "caar" "cadr" "cdar" "cddr"))
> -> "\\(?:c\\(?:\\(?:a[ad]\\|d[ad]\\|[ad]\\)r\\)\\)"
> First of all, there is an (apparently) unnecessary group around the result.
FWIW, I think this is not an error: we want (concat (regexp-opt STRS) "*")
to have a well-defined behavior (i.e. allow any number of repetitions of
> (regexp-opt '("master" "monster" "mister"))
> -> "\\(?:m\\(?:\\(?:on\\|[ai]\\)ster\\)\\)"
> It would be better (imo) eliminate unnecessary groups (those without "\\|")
> that the result was
Here, OTOH, the second (shy) subgroup is indeed unnecessary.
Regarding improving regexp-opt, in the general case you're looking at
minimizing finite state automatons. When regexp-opt was written, the
main purpose was to try and reduce backtracking and for that it's
perfectly sufficient to turn ("ack" "attack") into
"a\\(?:ck\\|ttack\\)". I later added "tail sharing" so that ("ack"
"attack") turns into "a\\(?:tta\\)?ck" but that's not really much use in
practice. We could try and get fancier, but it will tend to slow down
regexp-opt even more for rather small benefits (except in corner cases).
A much better approach is to go for a real "regexp to NFA/DFA
conversion". The `lex.el` package is one such example, but it's very
inefficient (in terms of building the FA and in the size of the FA, not
in terms of running the FA).