Difference between revisions of "Visual editor"

Revision as of 17:02, 14 March 2016

Links[edit | edit source]

Back story[edit | edit source]

Wiki markup is a sort of shorthand for more complicated HTML markup designed to create nicely formatted web documents. Although in the beginning wiki markup simplified the process of creating nicely formatted content, that shorthand can get increasingly complex in order to produce complicated layout and content. The extension of the markup system over time to include a template subsystem and macros inevitably leads to a tension between ease of use, and power. This tension was one of the primary factors that the Wikimedia Foundation cited as an impediment to new content collaborators for the Wikipedia. ^[1]^[2]^[3]. A visual editor was deemed nearly impossible after many early attempts failed, but with the advent of Node.js technology, it became an idea that was plausible again.

Intro[edit | edit source]

VIDEO Wikimania 2013
VIDEO (47 min.) with James Forrester - Product Manager and Roan Kattouw - Sr. Software Engineer presenting Jan 13, 2014 at Linux.conf.au on the contentEditable aspects of the system.

The Visual Editor for MediaWiki relies on the Parsoid extension ^[4] AND the Parsoid service. The Parsoid service is a node.js server that interacts with the mediawiki API.

"Parsoid implements a bidirectional wikitext parser and interpreter. It converts and interprets wikitext into an annotated HTML DOM, which can then be edited with HTML editor tools such as the Visual Editor. It also provides the conversion of a (possibly modified) HTML DOM back to wikitext."

Robin Cover would be proud. It round-trips wiki to HTML + RDFa and back.

Visual editor
Summary
Description:	A reliable rich-text editor for MediaWiki. Allows you to just "edit the page you see" while being aware of all the syntactically rich features of templates, extensions etc.
More
Notes:	The Visual Editor is available Here and on Wikipedia.
Test:	Login to the wiki, and make an edit in your personal sandbox.

Vendors[edit | edit source]

MyWikis is one of the few hosting providers that actually offers Visual Editor. Additionally, you can get a hosted parsoid service for $1/month if you want to run your own wiki with Visual Editor, but don't want to or can't setup the parsoid service. https://www.mywikis.com/hostedparsoid

Installation[edit | edit source]

You'll need at least MediaWiki v1.23. This site usually runs the most recent version available. We're using 1.34.4 (e34e7f2) at the moment. See what version that they're running on mediawiki.org and check yours too for free at https://freephile.org/wikireport

See the new RESTBase caching proxy system usable by VE

WebFaction[edit | edit source]

To install the Visual Editor extension in a Webfaction account, you need to first have node installed and working. You can do this by using the installer in your webfaction panel and associating a path with the app for a domain. I chose to install 'node' (which is now it's own one-click install). Once you do that you can visit the path you configured for node and you should see the "hello world" app that the installer puts in place. You'll need to stop that app, plus remove the cron job that respawns it. Then, in order to reserve and associate a port for the service, you install another 'app'. You do not need to open this port in the firewall unless you plan to share it with other instances, since HTTP talks to the node server on the backend. (I had to export the PORT, but this is no longer necessary if you configure it in localsettings.js ^[5].) Depending on the resource constraints of your hosting package, you may need to limit the number of workers spawned by the Parsoid service. It calculates a number based on the number of processors. For a shared hosting account, this is guaranteed to exceed your RAM or CPU limits, so specify the --num-workers option unless you're running on a dedicated host or VM.

Follow the installation instructions, and remember to edit both LocalSettings.php and localsettings.js When you setup the "interwiki" link to point to your API endpoint, you can choose pretty much whatever name you want to give it. The example is 'localhost'.

You'll want to set/update your $NODE_PATH and $PATH environment variables to locate the installed node modules, and to find the node and npm binaries

export NODE_PATH="$HOME/webapps/node/lib/node_modules:$NODE_PATH"
echo 'export NODE_PATH="$HOME/webapps/node/lib/node_modules:$NODE_PATH"' >> $HOME/.bashrc
which npm
export PATH="$HOME/webapps/node/bin:$PATH"
echo 'export PATH="/$HOME/webapps/node/bin:$PATH"' >> $HOME/.bashrc

Once node (and npm) is available, you of course need to get the parsoid and visual_editor extensions. A quick npm install in the parsoid directory sets up all the node machinery.

To have it run continuously, you might want to use the 'forever' module:

npm install -g forever

But that is just a node way of monitoring processes. You might want to simply use tools built for the purpose like tmux or screen plus monit. If using forever, then you just prefix your call to node with 'forever'

At this point you're ready to run the node server. Note: while installing this, I found a bug which prevented the server from listening on the configured port. It was awesomely cool that I mentioned the bug in IRC #mediawiki-parsoid and it was fixed before I was done testing my installation!!

cd ~/webapps/wiki/extensions/Parsoid/
node api/server.js --num-workers=2 &

The console should print out a list of worker pids, and the port number that the server is listening on. In your web browser, if you visit that path and port (setup in your WebFaction node mapping), you should see the Parsoid web service API message.

There is no stop script for a shared host, but you should be able to do something like

for pid in $(/sbin/pidof `which node`); do kill -9 $pid; done

That's because node is installed under your user account, and the $PATH is set as above to find that instance of node. To be sure that you're not going to kill everyone's node instances, you could look at all the nodes with ps -ef

Info[edit | edit source]

https://docs.webfaction.com/software/nodejs.html

Digital Ocean[edit | edit source]

Setup the Parsoid service
Set the /etc/mediawiki/parsoid/settings.js file to API endpoint (https://freephile.org/w/api.php)
sudo service parsoid restart
sudo tail -f /var/log/parsoid/parsoid.log

Note that a Visual editor setup (Parsoid service) includes configurations outside of your wiki installation (in /etc and maybe /usr/lib) so you'll want to preserve those files when doing host migrations.
See here

Extensions[edit | edit source]

API / Developer Docs / DevOps[edit | edit source]

mw:Parsoid/Setup
mw:Parsoid/Troubleshooting
mw:Parsoid/Debugging
mw:Parsoid/Packaging
DevOps (Wikitech)
https://doc.wikimedia.org/VisualEditor/master/ powered by JSDuck
http://blog.gmane.org/gmane.science.linguistics.wikipedia.wikitext wikitext-l

I noticed a bug and it was fixed about 5 minutes after I mentioned it! (Thanks C. Scott Ananian)

References[edit source]

↑ https://strategy.wikimedia.org/wiki/March_2011_Update
↑ http://strategy.wikimedia.org/wiki/Editor_Trends_Study
↑ See this October 2013 article in the MIT Technology Review http://www.technologyreview.com/featuredstory/520446/the-decline-of-wikipedia
↑ Parsoid extension
↑ parsoidConfig.serverPort = 8080 if you are running MediaWiki-Vagrant

[1] ttps://strategy.wikimedia.org/wiki/March_2011_Update

[2] ttp://strategy.wikimedia.org/wiki/Editor_Trends_Study

[3] See this October 2013 article in the MIT Technology Review http://www.technologyreview.com/featuredstory/520446/the-decline-of-wikipedia

[4] Parsoid extension

[5] parsoidConfig.serverPort = 8080 if you are running MediaWiki-Vagrant

[1]

[2]

[3]

[4]

[5]

@@ Line 1: / Line 1: @@
-{{Feature
-|image=Visual Editor (linking).png
-|imgdesc=Direct editing
-|title=
-}}
-{{#set:feature description = A reliable rich-text editor for MediaWiki.  Allows you to just "edit the page you see" while being aware of all the syntactically rich features of templates, extensions etc.}}
-{{#set:feature notes = The Visual Editor is available Here and on Wikipedia.}}
-{{#set:feature tests = [[Special:UserLogin|Login]] to the wiki, and make an edit in your [[Special:MyPage/Sandbox|personal sandbox]]. }}
-{{#set:feature examples = }}
 [[File:VisualEditor-logo.svg|alt=Visual Editor|frameless|641x641px|Visual Editor logo]]
 == Links ==
@@ Line 18: / Line 7: @@
 == Back story ==
 Wiki markup is a sort of shorthand for more complicated HTML markup designed to create nicely formatted web documents.  Although in the beginning wiki markup simplified the process of creating nicely formatted content, that shorthand can get increasingly complex in order to produce complicated layout and content.  The extension of the markup system over time to include a template subsystem and macros inevitably leads to a tension between ease of use, and power.  This tension was one of the primary factors that the Wikimedia Foundation cited as an impediment to new content collaborators for the Wikipedia. <ref>https://strategy.wikimedia.org/wiki/March_2011_Update</ref><ref>http://strategy.wikimedia.org/wiki/Editor_Trends_Study</ref> <ref>See this October 2013 article in the MIT Technology Review http://www.technologyreview.com/featuredstory/520446/the-decline-of-wikipedia</ref>.  A visual editor was deemed nearly impossible after many early attempts failed, but with the advent of Node.js technology, it became an idea that was plausible again.
-Fast-forward to 2016 when VisualEditor was fully working as a Visual WYSIWYG interface for editing MediaWiki content, using a Node.js backend called 'Parsoid' which is a bidirectional runtime wiki parser. This caused a big software architectural problem: there were two parsers to maintain: the 'normal' (now 'legacy') core parser in PHP, and the Visual Editor parser (Parsoid) in JavaScript. So, by the end of 2019, Parsoid was ported to PHP, and distributed with MediaWiki 1.35 as part of the [mw:Parsoid/Parser_Unification|Parser Unification initiative]].
 == Intro ==
@@ Line 25: / Line 12: @@
 * [https://www.youtube.com/watch?v=M_Ioi1aLAL8 VIDEO] (47 min.) with James Forrester - Product Manager and Roan Kattouw - Sr. Software Engineer presenting Jan 13, 2014 at Linux.conf.au on the contentEditable aspects of the system.
-The [http://www.mediawiki.org/wiki/VisualEditor Visual Editor] for MediaWiki relies on [[mw:Parsoid|Parsoid]]
+The [http://www.mediawiki.org/wiki/VisualEditor Visual Editor] for MediaWiki relies on the '''[[mw:Extension:Parsoid|Parsoid]]''' extension <ref>[https://git.wikimedia.org/tree/mediawiki%2Fextensions%2FParsoid Parsoid extension]</ref> AND the Parsoid service.  The Parsoid service is a node.js server that interacts with the mediawiki API.
 <blockquote>"Parsoid implements a bidirectional wikitext parser and interpreter. It converts and interprets wikitext into an annotated HTML DOM, which can then be edited with HTML editor tools such as the Visual Editor. It also provides the conversion of a (possibly modified) HTML DOM back to wikitext."</blockquote>
 [http://xml.coverpages.org/ Robin Cover] would be proud.  It round-trips wiki to HTML + RDFa and back.
+{{Feature
+|explains= Visual editor
+|description=A reliable rich-text editor for MediaWiki.  Allows you to just "edit the page you see" while being aware of all the syntactically rich features of templates, extensions etc. through a metadata system.
+|notes=
+|tests=
+|examples=
+}}
+== Vendors ==
+MyWikis is one of the few hosting providers that actually offers Visual Editor.  Additionally, you can get a hosted parsoid service for $1/month if you want to run your own wiki with Visual Editor, but don't want to or can't setup the parsoid service.  https://www.mywikis.com/hostedparsoid
 == Installation ==
-VisualEditor is included with MediaWiki. See https://www.mediawiki.org/wiki/Extension:VisualEditor
+You'll need at least [http://www.mediawiki.org/wiki/Release_notes/1.23 MediaWiki v1.23].  This site usually runs the most recent version available. We're using {{CURRENTVERSION}} at the moment. See what version that they're running on mediawiki.org and check yours too for free at https://freephile.org/wikireport
+See the new [[mw:RESTBase|RESTBase]] caching proxy system usable by VE
+=== WebFaction ===
+To install the Visual Editor extension in a Webfaction account, you need to first have node installed and working.  You can do this by using the [http://docs.webfaction.com/software/custom.html installer in your webfaction panel] and associating a path with the app for a domain.  I chose to install 'node' (which is now it's own one-click install).  Once you do that you can [https://freephile.org/node visit the path] you configured for node and you should see the "hello world" app that the installer puts in place.  You'll need to stop that app, plus remove the cron job that respawns it.  Then, in order to reserve and associate a port for the service, you install another 'app'.  You do not need to open this port in the firewall unless you plan to share it with other instances, since HTTP talks to the node server on the backend.  (I had to export the PORT, but this is no longer necessary if you configure it in localsettings.js <ref><code>parsoidConfig.serverPort = 8080</code> if you are running MediaWiki-Vagrant</ref>.)  Depending on the resource constraints of your hosting package, you may need to limit the number of workers spawned by the Parsoid service.  It calculates a number based on the number of processors.  For a shared hosting account, this is guaranteed to exceed your RAM or CPU limits, so specify the <code>--num-workers</code> option unless you're running on a dedicated host or VM.
-== Extensions ==
+Follow the installation instructions, and remember to edit both LocalSettings.php and localsettings.js  When you setup the "interwiki" link to point to your API endpoint, you can choose pretty much whatever name you want to give it.  The example is 'localhost'.
-# [[mw:Extension:VisualEditor]]
-# [[mw:Extension:Parsoid]]
+You'll want to set/update your $NODE_PATH and $PATH environment variables to locate the installed node modules, and to find the node and npm binaries
+<source lang="bash">
+export NODE_PATH="$HOME/webapps/node/lib/node_modules:$NODE_PATH"
+echo 'export NODE_PATH="$HOME/webapps/node/lib/node_modules:$NODE_PATH"' >> $HOME/.bashrc
+which npm
+export PATH="$HOME/webapps/node/bin:$PATH"
+echo 'export PATH="/$HOME/webapps/node/bin:$PATH"' >> $HOME/.bashrc
+</source>
+Once node (and npm) is available, you of course need to get the parsoid and visual_editor extensions.  A quick <code>npm install</code> in the parsoid directory sets up all the node machinery.
+To have it run continuously, you might want to use the 'forever' module:
+<source lang="bash">
+npm install -g forever
+</source>
+But that is just a node way of monitoring processes.  You might want to simply use tools built for the purpose like [[tmux]] or [[screen]] plus [[monit]].  If using <code>forever</code>, then you just prefix your call to node with 'forever'
+At this point you're ready to run the node server.  Note: while installing this, I found a bug which prevented the server from listening on the configured port.  It was awesomely cool that I mentioned the bug in IRC #mediawiki-parsoid and it was fixed before I was done testing my installation!!
+<source lang="bash">
+cd ~/webapps/wiki/extensions/Parsoid/
+node api/server.js --num-workers=2 &
+</source>
+The console should print out a list of worker pids, and the port number that the server is listening on.  In your web browser, if you visit that path and port (setup in your WebFaction node mapping), you should see the Parsoid web service API message.
-=== Extensions that hook into VE ===
+There is no stop script for a shared host, but you should be able to do something like
-* https://codesearch-beta.wmcloud.org/things/?q=ve.ui.commandRegistry.register&files=&excludeFiles=&repos=
+<source lang="bash">
-* https://codesearch-beta.wmcloud.org/extensions/?q=VisualEditorPluginModules&i=nope&files=&repos=
+for pid in $(/sbin/pidof `which node`); do kill -9 $pid; done
+</source>
-Extension authors should register <code>VisualEditorPluginModules</code>
+That's because node is installed under your user account, and the $PATH is set as above to find that instance of node.  To be sure that you're not going to kill everyone's node instances, you could look at all the nodes with ps -ef
-# https://www.mediawiki.org/wiki/Extension:Proofread_Page
+==== Info ====
-# https://www.mediawiki.org/wiki/Extension:CodeMirror (syntax highlighting)
+* https://docs.webfaction.com/software/nodejs.html
-# https://www.mediawiki.org/wiki/Extension:Disambiguator
-# https://www.mediawiki.org/wiki/Extension:Graph
-# https://www.mediawiki.org/wiki/Extension:Kartographer (Open Street Maps)
-# https://www.mediawiki.org/wiki/Extension:LanguageTool
-# https://www.mediawiki.org/wiki/Extension:Math
-# https://www.mediawiki.org/wiki/Extension:Score (LillyPond)
-# https://www.mediawiki.org/wiki/Extension:SyntaxHighlight
-# https://www.mediawiki.org/wiki/Extension:TemplateData
-# https://www.mediawiki.org/wiki/Extension:WikiHiero
+=== Digital Ocean ===
+# [https://www.mediawiki.org/wiki/Parsoid/Setup Setup the Parsoid service]
+# Set the <code>/etc/mediawiki/parsoid/settings.js</code> file to [[API]] endpoint (https://freephile.org/w/api.php)
+# sudo service parsoid restart
+# sudo tail -f /var/log/parsoid/parsoid.log
-== Customizing Visual Editor with custom menus, tools, commands ==
+* Note that a Visual editor setup (Parsoid service) includes configurations outside of your wiki installation (in /etc and maybe /usr/lib) so you'll want to preserve those files when doing host migrations.
-* https://www.mediawiki.org/wiki/VisualEditor/Gadgets
+* See [http://www.mediawiki.org/wiki/Thread:Extension_talk:VisualEditor/Digital_Ocean_Droplet:_Ubuntu_14.04_x64_with_MediaWiki_1.24.0 here]
-* https://www.mediawiki.org/wiki/VisualEditor/Gadgets/Creating_a_custom_command
-* https://www.mediawiki.org/wiki/VisualEditor/Gadgets/Add_a_tool
+== Extensions ==
+# [[mw:Extension:VisualEditor]]
+# [[mw:Extension:Parsoid]]
 == API / Developer Docs / DevOps ==
@@ Line 71: / Line 97: @@
 * https://doc.wikimedia.org/VisualEditor/master/ powered by JSDuck
 * http://blog.gmane.org/gmane.science.linguistics.wikipedia.wikitext wikitext-l
+I [https://gerrit.wikimedia.org/r/#/c/132479/1/api/ParsoidService.js noticed a bug] and it was fixed about 5 minutes after I mentioned it!  (Thanks C. Scott Ananian)
 {{References}}

Difference between revisions of "Visual editor"

Revision as of 17:02, 14 March 2016

Contents

Links[edit | edit source]

Back story[edit | edit source]

Intro[edit | edit source]

Vendors[edit | edit source]

Installation[edit | edit source]

WebFaction[edit | edit source]

Info[edit | edit source]

Digital Ocean[edit | edit source]

Extensions[edit | edit source]

API / Developer Docs / DevOps[edit | edit source]

References[edit source]

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Freephile

eQuality Technology

Categories

Print/export

Tools