Chess Engine Protocols (Dec 2002)

Introduction

Today we  take for granted the idea of engine communication protocols, and the ability to switch engines from GUI to GUI whether it is through the use of the open Xboard/Winboard protocol ,the newer Universal Chess Interface protocol, or the proprietary Chessbase protocol.

At the same time,we are faced with an explosion in Winboard and UCI chess engines. Not a day goes by without a dozen new versions of an existing engines being announced and we are blessed with the release of an average of 4-5 brand new chess engines per month. Maintainers of chess engines lists like Frank Quisinsky/Thomas Mayer in the past and now Leo Djiksman routinely update their sites  daily to keep up with the rapid pace.

But how and why did this development come about? The Bible tells the tale of  the Tower of Babel, where God cursed man with different tongues.This is roughly similar to the situation  before the rise of Xboard/Winboard communication protocol, where each Chess engine "spoke"  it's own language and could only work within  it's own interface. It  was probable that programmers didn't really think in such terms as chess engines and interfaces as both were tightly integrated.   Whenever someone wanted to build a chess program they had to code both the interface and engine from scratch.. A strict separation between the 2 was not necessary since the engine was not designed to be portable.

However, there was nothing inevitable about this, this article attempts to explain why the concept of protocols came about.We will first turn to the biggest and most successful protocol to date, the Xboard/Winboard protocol.

Interestingly enough, the development of the "Winboard protocol" was more a quirk of fate than of deliberate planning. According to Tim Mann (in a email interview with Frank Quisinsky, 2 April 2000) , at the time Xboard was built as the interface for Gnuchess and like most programs wasn't really designed to allow other Chess engines to be imported in. However, unlike most programs, the engine part, GNUchess and the interface Xboard were distinct enough to be considered 2 programs. When Internet Chess server (ICS) support with zippy was introduced in Sept 1993, suddenly programmers began to realise the possibility and advantages of porting their engines into Winboard and they began asking Tim for instructions on how to do so!

The rise in the Xboard/Winboard protocol

Why would programmers want to do that?

• Winboard had ICS support, which meant that any Chess engine that ran in Winboard could run automatically in Internet Servers. This was obviously very useful for testing. Today, commercial packages like Chess Assistant,Chess Vision,Chess partner,Arena etc and even Fritz 7 allow users to use Chess engines online, but back then, Xboard (and robofics) was the only and easiest way to get a Chess engine online. ¹
• The possibility and ease of engine matches on the same computer, now programmers could run automatically hundreds of automated games between 2 engines (either different versions of the same one, or against a different engine) ² .
• A starting Chess programmer, could just concentrate on working on the engine or Chess playing part, which most people consider interesting and not waste time working on the interface since Winboard could handle that.

The power of a common communication protocol had being discovered! And it was from those questions that the ad-hoc Winboard protocol was built up.

For the record Crafty was probably the first free Chess engine that supported winboard, By Jan 97 there were some 17 known free engines that supported the protocol, 50 by June 2000 and over 160 at the time of writing (year end 2002)

"The Winboarder effect"

Naturally like any standard, as more chess engines supported the standard the more useful it was to implement the standard. In addition to the above advantages, one unexpected advantage arose.

Due to efforts of webmasters like Djordje Vidanovic, (whose website on Winboard engines with tournaments and rating lists was probably one of the first) and Frank Quisinsky, a thriving community of experienced Winboard Chess engines users began to develop around his site and Volker Pittlik's Winboard web forum.

As a result, any amateur chess engine programmer, by implementing the Winboard protocol would be able to tap the expertise of these experienced users  (dubbed by some as "Winboarders") to help test their engine. ³

The effect fed on itself and By June 2000, there were 50 free Winboard engines.The 100th mark was reached in May 2000, and depending on how you figure it ⁴ , the honour belonged to Chezzz.

The commercials join in the act

By then, many began to take notice of the "Winboard effect" and by Feb 2000 many commercial software including Chessvision,Chess Assistant, Chess Partner and Chess Academy began to support the Winboard protocol (on top of their own) 

Finally in 2001, Chessmaster 8000 a product usually aimed at the mass market , began to support winboard engines .While Chessmaster was the last of the major chess commercial products to join in the party, it's support of Winboard was a milestone in Winboard history. For the first time, millions of users of the chessmaster product were introduced to the world of  Winboard engines.

While the commercial software support of the winboard protocol was carried out by and large smoothly without any fuss , one of the first commercial software to do so did it so poorly that it triggered off a long standing debate.

The Chessbase protocol

While the Xboard/Winboard movement was gathering steam, Chessbase, the makers of Fritz also saw (perhaps independently) the idea of allowing other Chess engines to be used within their interface. By June 1998, Fritz 5 supported the use of Junior 5 (fresh from it's victory at WMCC), HIRACS and NIMZO. By doing so, they probably had some of the best chess engines in their stable. However, unlike the open Xboard/winboard protocol,this  protocol was not open to all Chess engines, and the number of engines that could be used in Fritz was limited to those handful of engines.

However, in Dec 1998, Fritz 5.32 introduced the revolutionary (for it's time) feature of allowing winboard engines to be adapted into chessbase gui. It was hailed as a great step forward for Winboard engines as it meant that engines versus engine matches between top commercials and winboard engines was possible without resorting to frail setups with null serial cable modems auto232 players and Winboard auto232 adaptors .(Remi Coulom had released a auto232 adaptor for Winboard in Nov 1998)

Trouble with Chessbase's support of the Winboard protocol

Back then, there were 2 ways for a chess engine to run in chessbase. The first, was to submit the source to the chessbase people, who would then proceed to change the source to create a native engine that presumably ran on the chessbase protocol.This was the so called native version. This was done with Crafty which was a open source engine (similar but not exactly under the GNU license). The other which most people used (for various reasons, maybe because they didn't want to reveal their source), was to use the winboard adaptor.

Unfortunately within one month, it was discovered that there were serious flaws in the adaptor  The technical details are long and hard to detail,  in essence the chessbase gui sends a "new" command each turn to the engine, which makes it think there is a new game. It will then resend all the moves for the game up to that point.In addition there were a couple of oddities in the way it handled Winboard/Xboard engines ,with respect to odd moves,pondering, time problems etc.

This resending of moves (amusingly, the newer Universal Chess Interface protocol which will be covered later, does the same thing) weakens the adapted engine since it loses all the information stored in the hash tables up to then. There are other serious flaws as well, but this was the major sticking point

Through the years up to Fritz 6, chessbase refused to change the adaptor, despite complains from many chess engine authors and testers, who wanted a level playing field for winboard engines.

Despite years of debate, no one really knows why Chessbase has refused to fix this problem . One Chessbase representative even reportedly mentioned the bug could probably be fixed easily.One reason often stated was that the Winboard engines were meant merely as additional analysis engines (where they are presumably less or not weakened), and it was not anticipated that people would try to run engine versus engine matches in them. While some speculated darkly about conspiracies to weaken the performance of free Engines to make their own commercial engines look better in comparison, the truth might be simpler. It just wasn't worth the time or effort to fix the bug. Probably only a small percentage of their users even used the winboard adaptor (Crafty the most famous Chess engine was already available in the native version) and of these people even fewer were aware of the problem. And as for the hardcore computer chess addicts, they would buy Chessbase products regardless of what happened! So it was probably simple economics rather then intentional malice.

Of course, this all became moot, when in Jan 2002, Chessbase ( in a reorganisation of their webpages) , quietly dropped offering the Winboard adaptor for download without any explanation. At the same time, rumours began circulating that with Stefan Meyer-Kahlen  (author of Shredder and Universal Chess Interface Protocol) joining the chess team, Chessbase would begin to support the Universal Chess Interface (UCI) instead. This duly happened in Feb 2002, with a update for Fritz 7. More about UCI later.

The Winboard Protocol marches on

Interestingly enough, there was little or no trouble mostly with other commercials chess interfaces (exception seems to be the Shredder 5),

There were even attempts to sell commercial Winboard engines like Gandalf by Gambitsoft in Sept 2000. ⁵ This was a significant event, because for the first time a product was sold on the strength of it's engine and not the interface which was free. In theory, the consumer could pay less by just paying for the engine without the need to buy the same interface over and over again.For example, customers who bought Fritz 6, Junior 6 , basically paid for the same interface twice. ⁶

In the following years, there were attempts to improve the aging protocol, but due to the nature of the protocol which was mainly formed ad-hoc and the need to keep backward compatibility with older engines, progress was slow. In Dec 2000, Winboard 4.2.0 beta with support for the Winboard protocol II was released.

A minor fix it added a couple of commands requested by Chess engine programmers.To date, discussions are still going on for the third version of the protocol. For example changing the level command  to allow secondary time controls (now supported in Arena and Chessmaster 9000?) is one discussed improvement.

The UCI challenge

The Winboard protocol was not the only attempt at a communication protocol. In Nov 28,2000 Stefan Meyer-Kahlen began to publicise a new open free, Chess communication protocol known as UCI. ⁷

Like the winboard protcol it was free to use without license fees, but it had one drawback unlike Winboard which was a free interface, there was no free interface that supported UCI. At that time, the only interface that supported it was the commercial shredder 5.

A totally new protocol built from the scratch, it was not constrained to supporting older engines . It was built on a totally different principle from Winboard protocol. Unlike the latter, UCI was a stricter protocol that closely monitored the actions of the engine. While the author lists several advantages , some of them no longer apply to the newer Winboard protocols, and others have being dismissed as irrelevant or doable with the current protocols. The remaining were considered features that would be implemented in the winboard protocol III. For a semi-technical article on the standards war between Winboard and UCI refer to my article Winboard versus UCI the big debate

The rest of the article is a rehash of this article

One of UCI's greatest strength was that it was similar enough to Winboard protocol, that in theory anyone could convert a winboard compatible one to a UCI one quickly . In fact,some authors could use the same executable too support both protocols (though some had to use different ones).

Still progress was slow. Besides commercials engines like SOS,Shredder (by the authors of the UCI protocol), very few engine supported the standard. This was probably due to the lack in demand from users since there was only one interface that supported UCI (the next interface was the commercial chess assistant 6 in March 01), while dozens supported winboard protocols.

The UCI engine list was increased in May 2000, when Gambitsoft continued the experiment of selling stand alone engines, this time selling Winboard and UCI versions of Lambchop,Patzer,Capture and Gandalf (UCI only). A third series was planned but did not materialise, most of those (Pharaon,Dragon,Nejemet etc) were eventually released as UCI engines for free

There were few free UCI engines either (exceptions was Yace in Jan 2001, and later Pepito), From the amateur's point of view, there was little point in supporting UCI because the majority of users were not willing to buy a UCI interface (Shredder) just to help test a UCI amateur program.

Basically, it was a classic chicken and egg problem. No one would support UCI in their interface because there were few UCI engines and no one would implement UCI support in their engines because there was only one interface.

All this changed in Jan 2002, when Chessbase began to market Shredder .First, they released Shredder 6 in two versions a classic UCI version and a Chessbase version. But more importantly, perhaps because of constant complains of the winboard adaptor (as mentioned before) they announced that the patched version of Fritz 7 (and presumably all future Chessbase products) would support UCI while dropping the download of the adaptor from their site.

This alone gave a big boost to the popularity to UCI. After all Fritz and the Chessbase stable of products are probably the most popular interface for serious chessplayers. Competitors like Lokasoft/Rebel hastened to support UCI as well.

Chess engine authors also had the incentive to implement UCI, especially after it was found that UCI engines were not weakened as badly compared to the Winboard adaptor in Fritz 7 (or so they thought but see this for a description of a recently found serious problem ). Given that the World's strongest commercial engines Fritz,Junior,Hiarcs,and in 2001 Shredder,Tiger all ran in Chessbase protocol , this was the easiest way for Chess engine authors to get their chess engines running unweakened against these strong opponents.

In fact, the advantage of running chess engines in UCI mode compared to the Winboard mode was so great, that when Odd Malin's released his free wbitouci adaptor in April 2002 to adapt Winboard engines to UCI, many testers found that doing so led to better results than using the Winboard adaptor!

At the same time, Frank Quinsksy announced Arena by Marin Blume, .Arena was the first free interface to support both UCI and Winboard protocols.

Due to the twin effect, many Chess engines authors and users began to see reason to support UCI. Now that there was a free interface and a very popular commercial interface, the demand for UCI began to grow quickly. By March 2002 Chesspartner began to support UCI, this was followed by Gandalf 5.1 (the interface not engine) and Chess Academy 7 will follow suite by the end of 2002

In fact, since Jan 2002, besides the engines from the failed "Winboard Edition 3" package of UCI engines about 20 more UCI engines were released. For a total of 35 odd UCI engines (about 5 are commercial).

The future of Winboard and UCI

So which protocol will finally win out?What does the future hold?

In terms of functionality, UCI engines are easier to use because you can set common setting like Hash table size, endgame table base path etc for all UCI engines. There are also provisions within UCI to allow you to change settings within the interface.UCI also as provisions for showing hash table information. Other functions like multi-pv (or K-best lines) might also appeal to some.

On the other hand, some programmers have expressed concerns that the UCI mode is ugly or inelegant (it sends the whole move list each game), difficult to program, inflexible etc.. All this is probably a matter of taste.

In terms of user demand driving the use of the protocols, given that UCI now has both free and commercial interfaces available, Winboard no longer holds the upper hand.In fact, given the crippled Winboard support in Chessbase, UCI engines might have a slight edge here. for chess engine authors seeking to test their engines against the best Chessbase engines.

Still, I don't expect the Winboard protocol to become obsolete. In a review article on June 2002 , I made the prediction that while the number of UCI engines would continue to rise, this would not be at the expense of Winboard engines as existing and new engines would continue to support both. As of Dec 2002, it appears my prediction is still on track, of the 30 or so new UCI engines, only 5 do not support UCI.Of the 5, 2 are dedicated mate solvers, and only 1 is an existing Winboard engine which has dropped Winboard support (Goliath) and 2 (Silke Chess and Delphimax) are new engines with only UCI support. This is still heavily outweighed by existing Winboard engines adding UCI support and new engines supporting both protocols.

1. Much to the chagrin of FICS/ICC (and before that ICS) admins, this caused the online server to be flooded with GNUChess and later Crafty clones in 1997/8. This situation was repeated much later with Chess Tiger (by far the strongest commercial engine that could be used easily, since Fritz required the use of a combination of Winboard and Winboard auto232 adaptors plus a null modem cable) , flooding the servers, when Chess assistant/partner was released.

2. Of course,for a while it looked like auto232 players might become the de-fato standard given that many including Shredder 2,Fritz,Nimz0 98 supported it, but those were too difficult to use for the average user  and was often unstable

3. It appears the pendulum might have swung the other way.With over 150 engines,it's difficult to get excited over yet another new chess engine.There are now far too many engines for even the most devoted fan to keep track of all and inevitably, the weaker engines are ignored and get little help in testing . In recognition of this, Holmes - a mid-level engine by Andreas Herrmann has being withdrawn from public testing and is available only by email request from serious testers. (Discussion in Winboard forum)

4. I'm following Thomas's Mayer's counting method here. In Gunther Simon's list , the 100th engine is Pharaon.Here's a older list by Frank Quisinsky . For a full explanation of the reasons for this difference , see Appendix II

5. Technically, Gandalf was not the first commercial Winboard engine. Nimzo2000b was packaged as a Winboard engine as part of a package of Engines sold by Millennium Chess in Nov 99. Also Shredder 3, had a undocumented feature that allowed it to be used as a Winboard engine.

6. There was also a short lived attempt by Chessbase to sell a packet of Chess engine at reduced prices. Dubbed "Young talents" (mostly but not all Chessbase versions of Winboard engines). However there was no sign, they would do this for their first  rate Commercial engines. (Fritz,Junior etc)

7. After a first attempt,in 1998 with another new protocol MCS (Millennium Chess System) supported by Zarkov , Shredder 3, Wchess2000 and Chess Genius failed to take off