bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

Winston Weinert
I noticed EWW sends header "Accept: */*", which appears to prevent
HTTP servers that provide multiple content types from picking a sensible
response content type.

More information about the Accept header with a list of RFCs at the
bottom of the page can be found here [1]. For comparison lynx sends a
more specific header, which indicates It'd prefer common web content
types before anything else [2]. I experienced this particular bug when
browsing [3], which uses the Accept header to determine when to show the
built-in API Explorer. In a web browser it shows a HTML form to interact
with the API, however actual API clients use "Accept: application/json"
or similar to get the desired format.


Regards,
Winston Weinert


[1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept
[2]: "Accept: text/html, text/plain, text/sgml, text/css,
application/xhtml+xml, */*;q=0.01"
[3]: https://restframework.herokuapp.com/


In GNU Emacs 26.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll
bars)
  of 2018-09-08 built on snowcrash
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
System Description: Gentoo Base System release 2.4.1

Recent messages:
C-c C-g is undefined
Quit
Contacting host: duckduckgo.com:443
ImageMagick error: insufficient image data in file `/tmp/foo.ico' @
error/icon.c/ReadICONImage/375
Contacting host: localhost:8000
Making completion list... [3 times]
Note: file is write protected
Mark saved where search started
Quit
delete-backward-char: Text is read-only

Configured using:
  'configure --prefix=/usr --build=x86_64-pc-linux-gnu
  --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
  --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
  --localstatedir=/var/lib --disable-silent-rules
  --docdir=/usr/share/doc/emacs-26.1-r3
  --htmldir=/usr/share/doc/emacs-26.1-r3/html --libdir=/usr/lib64
  --program-suffix=-emacs-26 --infodir=/usr/share/info/emacs-26
  --localstatedir=/var
  --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
  --without-compress-install --without-hesiod --without-pop
  --with-file-notification=inotify --enable-acl --with-dbus
  --without-modules --without-gameuser --without-gpm --without-kerberos
  --without-kerberos5 --with-lcms2 --with-xml2 --without-mailutils
  --without-selinux --with-gnutls --without-libsystemd --with-threads
  --without-wide-int --with-zlib --with-sound=alsa --with-x --without-ns
  --without-gconf --without-gsettings --with-toolkit-scroll-bars
  --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --with-xpm
  --with-imagemagick --with-xft --without-cairo --without-libotf
  --without-m17n-flt --with-x-toolkit=lucid --with-xaw3d 'CFLAGS=-O2
  -pipe -march=bdver2' CPPFLAGS= 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS NOTIFY ACL
GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11 THREADS
LCMS2

Important settings:
   value of $LANG: en_US.utf8
   locale-coding-system: utf-8-unix

Major mode: Outline

Minor modes in effect:
   TeX-PDF-mode: t
   global-magit-file-mode: t
   diff-auto-refine-mode: t
   magit-auto-revert-mode: t
   global-git-commit-mode: t
   async-bytecomp-package-mode: t
   override-global-mode: t
   winum-mode: t
   shell-dirtrack-mode: t
   recentf-mode: t
   global-edit-server-edit-mode: t
   mode-line-bell-mode: t
   editorconfig-mode: t
   electric-pair-mode: t
   image-diredx-async-mode: t
   show-paren-mode: t
   cl-old-struct-compat-mode: t
   tooltip-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   buffer-read-only: t
   column-number-mode: t
   line-number-mode: t
   transient-mark-mode: t

Load-path shadows:
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim hides
~/.emacs.d/custom/emacs-eclim/eclim
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-java-run hides
~/.emacs.d/custom/emacs-eclim/eclim-java-run
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-macros hides
~/.emacs.d/custom/emacs-eclim/eclim-macros
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-debug hides
~/.emacs.d/custom/emacs-eclim/eclim-debug
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-completion hides
~/.emacs.d/custom/emacs-eclim/eclim-completion
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-common hides
~/.emacs.d/custom/emacs-eclim/eclim-common
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-problems hides
~/.emacs.d/custom/emacs-eclim/eclim-problems
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-maven hides
~/.emacs.d/custom/emacs-eclim/eclim-maven
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclimd hides
~/.emacs.d/custom/emacs-eclim/eclimd
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-ant hides
~/.emacs.d/custom/emacs-eclim/eclim-ant
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-pkg hides
~/.emacs.d/custom/emacs-eclim/eclim-pkg
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-project hides
~/.emacs.d/custom/emacs-eclim/eclim-project
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-scala hides
~/.emacs.d/custom/emacs-eclim/eclim-scala
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-java hides
~/.emacs.d/custom/emacs-eclim/eclim-java
~/.emacs.d/custom/emacs-eclim/company-emacs-eclim hides
/home/winston/.emacs.d/elpa/company-emacs-eclim-20180911.1121/company-emacs-eclim
~/code/cyberpunk-theme.el/cyberpunk-theme hides
/home/winston/.emacs.d/elpa/cyberpunk-theme-20180609.509/cyberpunk-theme

Features:
(shadow sort mail-extr emacsbug rng-xsd xsd-regexp rng-cmpct rng-nxml
rng-valid nxml-mode nxml-outln nxml-rap sgml-mode tabify man scheme
yaml-mode term ehelp sml-mode conf-mode gist gh-gist gh-oauth gh-api
logito gh-cache pcache gh-auth gh-url gh-profile timezone eieio-base
cus-edit url-file url-dired eww mm-url url-queue helm-dash helm
helm-source helm-multi-match helm-lib eclim-scala eclim-debug eclimd gud
eclim-java-run eclim-ant eclim-maven eclim-project eclim-java
eclim-problems yasnippet eclim-common popup arc-mode archive-mode
eclim-macros package-recipe-mode package-build-badges package-build
package-recipe skeleton ibuf-macs grep ibuf-ext ibuffer ibuffer-loaddefs
racket-unicode-input-method quail racket-debug racket-mode
racket-bug-report racket-collection racket-stepper racket-logger
racket-profile racket-imenu racket-edit racket-complete shr svg dom
racket-repl racket-common racket-indent racket-font-lock racket-ppss
racket-keywords-and-builtins tq racket-util racket-custom sh-script smie
executable tar-mode autoload radix-tree lisp-mnt mm-archive
network-stream starttls url-cache tmm haskell-doc inf-haskell
haskell-decl-scan haskell haskell-completions haskell-load
haskell-commands highlight-uses-mode haskell-modules haskell-sandbox
haskell-navigate-imports haskell-repl haskell-svg haskell-collapse
hideshow haskell-debug haskell-interactive-mode
haskell-presentation-mode haskell-compile haskell-hoogle haskell-process
haskell-session vc-git haskell-mode haskell-cabal haskell-utils
haskell-font-lock haskell-indentation haskell-string
haskell-sort-imports haskell-lexeme haskell-align-imports haskell-compat
haskell-complete-module haskell-ghc-support etags xref project dabbrev
haskell-customize ssh-config-mode misearch multi-isearch preview
prv-emacs tex-buf font-latex latex tex-ispell tex-style tex dbus xml
tex-mode latexenc org-table ispell org-rmail org-mhe org-irc org-info
org-gnus nnir gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap
nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range
gnus-win org-docview org-bibtex bibtex org-bbdb org-w3m linum whitespace
editorconfig-core editorconfig-core-handle editorconfig-fnmatch
smart-mode-line-dark-theme smart-mode-line rich-minority cyberpunk-theme
magithub magithub-dash magithub-notification magithub-orgs
magithub-issue-tricks magithub-issue-post magithub-edit-mode
magithub-repo magithub-ci magithub-issue magithub-label magithub-user
magithub-core magithub-faces magithub-settings parse-time markdown-mode
color bug-reference ghub+ apiwrap apropos s magit-bookmark
magit-obsolete magit-blame magit-stash magit-bisect magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-collab ghub-graphql treepy
graphql pcase ghub url-http tls gnutls url-gw nsm url-auth url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util mailcap let-alist json map magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
diff-mode magit-core magit-autorevert autorevert filenotify
magit-process magit-margin magit-mode git-commit magit-git magit-section
magit-utils crm subr-x magit-popup log-edit pcvs-util add-log
with-editor async-bytecomp async server use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
use-package-core cl-extra winum dash python-mode which-func imenu shell
hippie-exp flymake-proc flymake warnings compile cc-cmds cc-engine
cc-vars cc-defs web-mode disp-table dashboard dashboard-widgets recentf
tree-widget page-break-lines org-agenda bookmark pp smex edit-server
keychain-environment help-mode dired+ image-file dired-x dired-aux
buffer-move windmove mode-line-bell editorconfig ox-twbs ox-odt rng-loc
rng-uri rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util ox-latex ox-icalendar ox-html table ox-ascii
ox-publish ox profiler org-element avl-tree generator org org-macro
org-footnote org-pcomplete pcomplete org-list org-faces org-entities
noutline outline easy-mmode org-version ob-emacs-lisp ob ob-tangle
org-src ob-ref ob-lob ob-table ob-keys ob-exp ob-comint ob-core ob-eval
org-compat org-macs org-loaddefs find-func elec-pair irfc scribble
finder-inf edmacro kmacro gh-common marshal eieio-compat image-dired+
image-dired rx pydoc-info advice info-look info package url-handlers
url-parse url-vars wheatgrass-theme paren gnus nnheader wid-edit
cus-start cus-load site-gentoo mu4e desktop frameset mu4e-speedbar
speedbar sb-image ezimage dframe mu4e-main mu4e-context mu4e-view
cal-menu calendar cal-loaddefs thingatpt browse-url comint ansi-color
ring mu4e-headers mu4e-compose mu4e-draft mu4e-actions ido rfc2368
smtpmail auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
sendmail mu4e-mark mu4e-message html2text mu4e-proc mu4e-utils doc-view
jka-compr image-mode mu4e-lists mu4e-vars message rmc puny seq byte-opt
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
easymenu mml-sec password-cache epa derived epg epg-config gnus-util
rmail rmail-loaddefs time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev
mail-utils gmm-utils mailheader hl-line cl gv cl-loaddefs cl-lib
mu4e-meta preview-latex auto-loads tex-site mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind inotify lcms2 dynamic-setting font-render-setting
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 2584773 148802)
  (symbols 48 78658 1)
  (miscs 40 5277 2890)
  (strings 32 395278 13798)
  (string-bytes 1 15855316)
  (vectors 16 114301)
  (vector-slots 8 2574679 85614)
  (floats 8 656 957)
  (intervals 56 273185 3106)
  (buffers 992 129))



Reply | Threaded
Open this post in threaded view
|

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

Federico Tedin
> I noticed EWW sends header "Accept: */*", which appears to prevent
> HTTP servers that provide multiple content types from picking a sensible
> response content type.
>
> More information about the Accept header with a list of RFCs at the
> bottom of the page can be found here [1]. For comparison lynx sends a
> more specific header, which indicates It'd prefer common web content
> types before anything else [2]. I experienced this particular bug when
> browsing [3], which uses the Accept header to determine when to show the
> built-in API Explorer. In a web browser it shows a HTML form to interact
> with the API, however actual API clients use "Accept: application/json"
> or similar to get the desired format.

I've been looking into this problem. It seems that EWW uses
url-retrieve to retrieve HTTP contents, which then calls url-http.  In
url-http, the variable url-mime-accept-string is used as the value for
the HTTP Accept header (if it is set to nil, "*/*" is used
instead). By default, url-mime-accept-string is set to nil.

We could temporarily set the variable url-mime-accept-string with
another value when url-retrieve is called by EWW. I am not sure what
the default value should be, or if it should be configurable. Maybe
the same value used by Lynx would be a good starting point.