Issues creating search plugins using Mycroft Project plugins
Some people mentioned using submitted search plugins from Mycroft Project as a template. Pick one for your search site (I was using Startpage.com).
It should be easy. It was easy - before. This time it wasn't - but did eventually work (long, long time). In the screen, you see under Post Parameters. I was using https://startpage.com and changing a few options. They create a URL with those settings, but they're no longer in json format. That may be the problem.
One result is in "plain text", but you'll see it's not json format. The other URL, the custom settings are obfuscated. I saw / tested other plugins for startpage from Mycroft that used the obfuscated string in BOTH the Post Parameters & Search Form URL entry & they seemed to worked fine. When I created mine & installed (them, about 5 diff times), it wouldn't start up with the color theme selected & saved in the string.
The only way I got that to work was look at some other plugins on Mycroft & after the "query={searchTerms}" , I had to add "&t=dark", or else once installed, on the 1st use from the search manager, it opened the site w/ light theme. After another search, it recognized the dark theme chosen (& saved in the string their settings page created).
Needless to say, there isn't any "&t=dark" setting in the strings that Startpage NOW creates. I used it because other plugins used it - even some recent ones w/ the "new" Startpage icon (2 blue rectangles w/ white stripe between). But there are no json strings being created by the site anymore for custom settings.
I'm guessing that the open search form / specs like on Mycroft Project need json format data? Because the custom settings string also had 20 results per page & it only shows the default of 10. I'd guess the site ignores the settings in the plugin that don't use json? Maybe some of the older plugins using obfuscated strings were actually json to begin with? I don't see how the site could read (json) data if it's obfuscated.
It appears their site still recognizes the names / values in json format, if you know the correct ones. I'd have to look in a lot of Startpage plugins to find the correct json format abbreviations for various settings, if that's the only way an open search plugin will work. Mozilla is making it hard for people not to use the included search engines - as is - for obvious reasons.
Modified by JoeB
All Replies (4)
You can see Startpage's default OpenSearch plugin parameters by pasting this URL to the address bar:
view-source:https://www.startpage.com/en/opensearch.xml
It is XML, not JSON. I'm not sure where you see JSON?
In your screenshot, you circled the .pl suffix. That usually indicates a Perl script, although of course it could be anything.
I think there is an effort underway to create an API for extensions to add new search plugins, and perhaps that will open the door to easier builders. I don't know how soon that is expected to "land" in testing.
Joebt said
... Mozilla is making it hard for people not to use the included search engines - as is - for obvious reasons.
Major changes to how Firefox handles Search plugin XML files happened a couple of years ago; I forget which exact version number brought the changes (iirc it was Fx 48) but it really screwed up how I was using & editing Search plugin XML files.
Now the Search plugin XML files get written to the search.json.mozlz4 file in the Profile folder at the time those changes took place (or the update procedure when the change landed) and when each new Search plugin gets installed. Any editing the user does after installation (or after that update occurred) doesn't edit the search.json.mozlz4 file.
See if this extension helps you: https://addons.mozilla.org/en-US/firefox/addon/mozlz4-edit/
Or instructions for using it without installation: https://github.com/serj-kzv/mozlz4-edit#using-without-the-installation
Or this: https://gist.github.com/Tblue/62ff47bef7f894e92ed5 - but it uses a Python 3 script which is far beyond my skill level.
Thanks for the replies. Some will be useful. I've got some mozlz4 decryption scripts, but not sure they still work. I'm too tired to re-write my response, that was lost.
I need to find all the parameter "short names" & possible values, not just default ones. When I used a customized bookmark - that holds settings changes, to open startpage & do a search, then look at page source, It had surprisingly little about the settings used (non-default). Didn't list all of them I changed when making the bookmark.
I've got to find the correct POST parameters for Startpage, DDG (I think they publish a LONG list) and Searx.me - for now. I have to say after looking at plugins on Mycroft (I've edited search plugins several times), I'm not sure some people know what they're doing. But, the "detailed" instructions leave a lot of unanswered questions on what goes where.
I don't think you can use the plain text form of settings or the obfuscated string created by Startpage in the XML files (not under POST Parameters. But a lot of plugins I looked at did that.
When I installed some to test, they didn't use some settings the plugins claimed to use. Until I moved some parameters to before the obfuscated string under Post Parameters, indicating the form doesn't recognize the obfuscated strings or the plain text style that Startpage generates.
Now we're cooking w/ gas, Edmeister. I love opening stuff to see what's inside. Everyone: MAKE A BACKUP OF EXISTING "search.json.mozlz4" compressed file, that contains all of your currently installed search plugins.
The mozlz4-edit.zip file that you mentioned does indeed work as "portable" - no installation (not a big deal in Linux).
Here's the format - inside the compressed file for an engine's custom settings. Note: I'm pretty sure there're no standard abbreviations for the names of settings between engine "brands." This is how Startpage now shows their Plain Text settings (meant to use as a custom bookmark). Some settings in the string are default & some custom. https://www.startpage.com/do/mypage.pl?prfh=suggestionsEEE1N1Ngeo_mapEEE0N1Nwikipedia_iaEEE1N1Nother_iaEEE1N1Ndisable_open_in_new_windowEEE0N1Ndisable_video_family_filterEEE1N1Nenable_post_methodEEE1N1Nenable_proxy_safety_suggestEEE1N1Ndisable_family_filterEEE1N1NsslEEE1N1Nlanguage_uiEEEenglishN1NlanguageEEEenglishN1Nwt_unitEEEcelsiusN1Nnum_of_resultsEEE20N1Nlang_homepageEEEs/dark/en/
This is how part of custom settings look once search.json.mozlz4 is decompressed w/ the addon the edmeister mentioned: "_urls": [
{
"template": "https://www.startpage.com/do/search",
"rels": [],
"resultDomain": "www.startpage.com",
"method": "POST",
"params": [
{
"name": "query",
"value": "{searchTerms}"
},
{
"name": "t",
"value": "dark"
},
{
"name": "limit",
"value": "20"
}
Under Params (parameters), name: t is their shortcut for "theme" with value : dark (one choice). That is obviously the correct shortcut for "theme," as this plugin used the dark theme. The shortcut name : "limit" is obviously not what Startpage now uses for "number of results per page", as this plugin only showed the default 10 results (for Startpage). I sent their support a request for a list of parameters, but could take weeks (sometimes has).
What I (think) I discovered - about Startpage (not necessarily other engines) is if you choose your custom settings, then they generate the obfuscated URL containing those settings. If you know how & where to enter that long URL string in the Mycroft Project plugin submission forms, it apparently works correctly & uses the chosen settings from the obfuscated string.
How do I know? I started with a submitted Startpage plugin - opened by its ref # by hovering the engine name in the search results page. Open this in another tab: https://mycroftproject.com/submitos.html, then enter the Ref. # of the submitted plugin in the "Load" field & it loads the data of that plugin.
If you want to use the POST method for searches (your search terms are not sent to pages you click from the search engine), in the POST parameters field, "query={searchTerms}" seems to be the minimum required. I followed what others did (for Startpage plugins) & entered ONLY PART of the obfuscated URL, that starts with: " prfe=36c845...", after the query={searchTerms}&prfe=36c845...
You must separate each term using ampersand '&'. Under the Search Form URL field, others seem to enter the entire obfuscated URL: https://www.startpage.com/do/mypage.pl?prfe=36c845.... And in the search.json file, under parameters, it shows; "name": "prfe", [yes, they're all in quotes & a comma after the name]
"value": "36c845..."
My custom settings from the obfuscated settings string overrode some settings already entered in the plugin I was using as a template. That's how I know the Startpage site reads the obfuscated string in the XML file, once the plugin is added to Firefox.
And doing a search with the newly added plugin, then viewing Startpage's Settings (I think java script must be allowed), it showed all custom settings I chose, that it included in the obfuscated URL. Anyone that's never used the Open Search form to create a plugin (or modify one like I did) will have a bit of reading to do. Looking at several completed plugins - opened in the */submitos.html form - gives an idea how others created plugins for specific engines.
Obviously, this is over the heads of users that only care to know how to open & close a browser - which is fine. That's why I said Mozilla's making it hard, UNLESS a search engine allows installing customized plugins from their site (many did pre-Fx Quantum).
One last thought. I don't know how much additional data, if any, a destination site could gather because you use a customized search engine. Maybe not much if using the POST method & also using search engines that don't log your IPa or search terms, like Startpage, DDG, Searx.me. If you use Google, Yahoo, Bing & others' to search - well... they're gathering a whole lot more than search engine settings.
Modified by JoeB