Arianne Development Meeting 2016-01-06: Difference between revisions
Jump to navigation
Jump to search
Content deleted Content added
imported>Hendrik Brummermann No edit summary |
imported>Hendrik Brummermann No edit summary |
||
| (22 intermediate revisions by the same user not shown) | |||
| Line 3: | Line 3: | ||
== Sourceforge, Github, Travis == |
== Sourceforge, Github, Travis == |
||
In August after an extended downtime of Sourceforge, we created the Arianne organisation on Github and mirrored the Stendhal git repository. Back than the mirroring was only from Sourceforge to github, and it was done by the testserver build script once a night. |
|||
* A quick update from Hendrik about the recent infrastructure changes |
|||
* ... |
|||
Since then we have improved the situation significantly: The mirroring is now done in both directions, and it is done almost instantaneously. This means it does not matter anymore whether commits are done to Sourceforge or Github. So we can use Github features such as pull request. And we can make use of Travis for running the JUnit test. [[Two way git mirror]] explains the details. |
|||
<!-- |
|||
20:06hendrik1. Recent infrastructure changes: Sourceforge, Github, Travis |
|||
In addition to Stendhal, Marauroa is now on github, too. |
|||
20:07hendrikIn August after an extended downtime of Sourceforge, we created the Arianne organisation on Github and mirrored the Stendhal git repository. |
|||
20:07hendrikBack than the mirroring was only from Sourceforge to github, and it was done by the testserver build script once a night. |
|||
All JUnit tests for Marauroa [https://travis-ci.org/arianne/marauroa] and Stendhal [https://travis-ci.org/arianne/stendhal] are passing. So before we do a new release, we can just check there. In addition, Travis will execute the test suite for each pull request and post the result directly to the pull request page. Unfortunetelly there is nice output of failed tests. So in case of failures, we have to look at the console log or execute the tests locally. |
|||
20:07hendrikSince then we have improved the situation significantly: |
|||
20:08hendrikThe mirroring is now done in both directions, and it is done almost instantaniously. |
|||
Failed tests are reported to IRC again. We should add the [https://travis-ci.org/arianne/stendhal.svg build-passing image] and Github to [https://arianne-project.org https://arianne-project.org] and [[Stendhal Testing]]. |
|||
20:08hendrikThis means it does not matter anymore whether commits are done to Sourceforge or Github. So we can use Github features such as pull request. And we can make use of Travis for running the Junit test. |
|||
20:09hendrikhttps://stendhalgame.org/wiki/Two_way_git_mirror explains the details. |
|||
We should document pull requests on [[How To Create Patch For Stendhal]]. We can recommend pull requests, but accept patches as well. For those familiar with Github pull requests are easy. But patches are simpler for the others. |
|||
20:09hendrikIn addition to Stendhal, Marauroa is now on github, too. |
|||
20:10hendrikAll junit tests for Marauroa https://travis-ci.org/arianne/marauroa and Stendhal https://travis-ci.org/arianne/stendhal are passing. |
|||
20:11alberthjavierhi |
|||
20:11hendrikSo before we do a new release, we can just check there. In addition, Travis will execute the test suite for each pull request and post the result directly to the pull request page. |
|||
20:12hendrikUnfortunetelly there is nice output of failed tests. So in case of failures, we have to look at the console log or execute the tests locally. |
|||
20:12hendrikSo far a quick infrastructure updates. Any comments or questions? |
|||
20:12kiherucan it notify about failure to irc like the old jenkins setup? |
|||
20:13hendrikAccording to https://docs.travis-ci.com/user/notifications/#IRC-notification that is supposed to work. I tried to configure it, but it does not work. |
|||
20:14hendrikhttps://github.com/arianne/stendhal/blob/master/.travis.yml |
|||
20:14kiheruaw :-( |
|||
20:14markusFirst of all I like travis and the new possibilities of github, esp. pull requests. This brings up a thing we could consider: pointing new contributors to creating pull requests instead of patches? |
|||
20:15hendrikI am not sure, if I made a mistake or whether it does not work anymore. In the second case, that's a task for postman. |
|||
20:15hendrikmarkus, yes, that may be a good idea. |
|||
20:16markusThe question would be: Do we think it is easier for them to create a patch or create a pull request from a fork |
|||
20:17markushendrik: didn't you tell me about uploading the testresults somewhere? |
|||
20:17hendrikYes, but that only works partly. |
|||
20:18markusah, okay. I thought about doing sth like our junit history at work |
|||
20:18hendrikBut it would be the place to invoke postman from. |
|||
20:18balaurI also like the idea of pull requests more, but for a patch you don't need an account |
|||
20:18hendrikYes, that was the idea. But that's quite a bit of work. And it's kind of boring having to develop that again from scratch. So I postponed it. |
|||
20:19hendrikPerhaps we should point out both alternatives to new contributors. Or would that be too confusing? |
|||
20:19kiheruWe can recommend pull requests, but accept patches as well. For those familiar with github pull requests are easy, but for those who arent, patcvhes are likely simpler |
|||
20:19*hendrik nods. |
|||
20:19markusyes, kiheru. |
|||
20:20balauri agree |
|||
20:20hendrikMarkus is refering to our testresults datenbank at work. It basically consists of a table of test runs, a table of test names, and a table of results. The result table links the other two tables. |
|||
20:21markusso we should add that to https://stendhalgame.org/wiki/How_To_Create_Patch_For_Stendhal as alternative path. |
|||
20:21markusI am thinking about possibities for jenkins to be abused for that. |
|||
20:22hendrikSo we can make database queries such as: Which tests failed this time, but were fine on the last run? Or: On which test run did the failure straigh start for this test? |
|||
20:22hendrikWe have no jenkins. |
|||
20:23markusI know :) But: If we have a "publish-only-reports"-style job, such a jenkins would not waste too much resources |
|||
20:24hendrikThis has rather low priority for me, because I assume that we will fix failing tests quickly. Not keep a large number of test unfixed for a long time. |
|||
20:24markusJust an idea to get there. But maybe needs more thinking. :) |
|||
20:25markusRegarding travis: Can we get the build status prominent in some place? |
|||
20:26hendrikhttps://travis-ci.org/arianne/stendhal.svg |
|||
20:26markusI was more aiming on embedding that badge in one of our pages |
|||
20:27hendrikWhich page? |
|||
20:28markusarianne-project.org maybe? |
|||
20:28hendrikhttps://stendhalgame.org/wiki/Stendhal_Testing ? |
|||
20:28hendrikOn the left site? |
|||
20:28markusyes, good idea :) |
|||
20:29hendrikAnd a github logo. |
|||
20:29markusyes. |
|||
20:29kiherucan't hurt to have it there. not as good as screaming about failures at irc, but at least a place where somebody will notice it hopefully |
|||
20:30hendrikI think it is "both" not "one or the other". |
|||
20:30hendrikOn the website it's basically advertisment that we are doing good work[tm]. |
|||
20:31hendrikOn IRC is were we will need it for practical reasons. |
|||
20:31markusexactly. :) |
|||
20:32kiheruagreed. hopefully the irc notification is something that can be fixed |
|||
20:33kiheruhey |
|||
20:33hendrikIt might be a good idea, if someone else looks at my configuration, to check that I did not mess it up. |
|||
20:33kiheruthre travis docs have "irc:" under section "notifications:" |
|||
20:34kiheruwhereas on the .travis.yml it is top level, if I read it right |
|||
20:35kiheru(I have been trying to find the problem during the discussion. that's the best idea yet) |
|||
20:35postmanarianne_rpg: nhnb * rc4c927c49d7c stendhal/.travis.yml: fixed travis irc configuration |
|||
20:35hendrikOops. |
|||
20:35kiheruI hope it worked |
|||
20:35balaurok now let's break a test :p |
|||
20:36hendrikShall we move to the next topic? |
|||
20:37travis-ciarianne/stendhal#102 (master - c4c927c : Hendrik Brummermann): The build passed. |
|||
20:37travis-ciChange view : https://github.com/arianne/stendhal/compare/a0d70ed2e3f1...c4c927c49d7c |
|||
20:37travis-ciBuild details : https://travis-ci.org/arianne/stendhal/builds/100660289 |
|||
20:37kiheruwoo! |
|||
20:37markusyay! |
|||
20:37hendrikkiheru, yay!!! |
|||
20:38postmanarianne_rpg: nhnb * r98affef72de0 stendhal/.classpath: report only test status changes |
|||
20:38*hendrik closes Eclipse. |
|||
--> |
|||
== Prepaing release of Marauroa 4.0 == |
== Prepaing release of Marauroa 4.0 == |
||
A couple of years ago we started to experiment with web clients. We did several attempts using different technologies. To cut a long story short: Only recent browser developments enabled us to build a working webclient. All this time, most of Marauroa's development was done on a branch named perception_json, with only minor changes being backported to master. |
|||
* After years of development, perception_json was merged into master. |
|||
* Release bockers are tagged with [https://sourceforge.net/p/arianne/developers/search/?q=labels:%22MARAUROA_03_99%22 MARAUROA_03_99] |
|||
Without a real working client, I was afraid that the Marauroa interface were not tested against real requirement. But the last attempt to write a Stendhal webclient is very promissing (although there is still a lot of work required). So perception_json branch was merged into master recently. |
|||
* ... |
|||
<!-- |
|||
We should do a Marauroa release in the near future: Release bockers are tagged with [https://sourceforge.net/p/arianne/developers/search/?q=labels:%22MARAUROA_03_99%22 MARAUROA_03_99] |
|||
20:39hendrik2. Marauroa 4.0 |
|||
20:40hendrikA couple of years ago we started to experiment with web clients. We did several attempts using different technologies. To cut a long story short: Only recent browser developments enabled us to build a working webclient. |
|||
* Most important is updating documentation and examples. |
|||
20:40hendrikAll this time, most of Marauroa's development was done on a branch named perception_json, with only minor changes being backported to master. |
|||
* Stendhal should be able load the maps from transfer-content so that that part of the json protocol gets tested as well. Currently Stendhal uses Marauroa's binary serialization internally to provide content. So either the JavaScript needs to understand it, or the server needs to provide alternative content |
|||
20:41travis-ciarianne/stendhal#103 (master - 98affef : Hendrik Brummermann): The build passed. |
|||
* Changing structure of gameEvents table to use references instead of source string. Postponed after 4.0 |
|||
20:41travis-ciChange view : https://github.com/arianne/stendhal/compare/c4c927c49d7c...98affef72de0 |
|||
* Re-adding support for Marven which was deleted because mvn test is broken and Travis insists on using Maven instead of Ant. We should aim to restore it before release. But this is not a release blocker. |
|||
20:41travis-ciBuild details : https://travis-ci.org/arianne/stendhal/builds/100661020 |
|||
20:41*hendrik glares at travis: That was not suppose to happen. But it has to wait until later. |
|||
20:42kiheruwhy not? the config says "on_success: always" |
|||
20:42hendrikI changed that with the recent commit to "change2. |
|||
20:43kiheruprobably not. you changed stendhal/.classpath, not stendhal/.travis.yml |
|||
20:43hendrikargh. |
|||
20:44hendrikWithout a real working client, I was afraid that the Marauroa interface were not tested against real requirement. |
|||
20:44hendrikBut the last attempt to write a Stendhal webclient is very promissing (although there is still a lot of work required). |
|||
20:44hendrikSo I merged the perception_json branch into master, updated CHANGES.txt. |
|||
20:45hendrikI think we should do a Marauroa release in the near future. |
|||
20:45hendrikhttps://sourceforge.net/p/arianne/developers/search/?q=labels:%22MARAUROA_03_99%22 |
|||
20:46hendrikThat is a list of tickets concerning aspects that we should think about before releasing. |
|||
20:46hendrikMost important is updating documentation and examples. |
|||
20:46kiheruprobably should try to make the webclient load the maps from transfercontent so that that part of the json protocol gets tested as well |
|||
20:48*hendrik nods: That is one major remaining problem: Stendhal makes use of Marauroas transfer protocol. |
|||
20:48hendrikThat is, at least some layers such as tileset and map_data, contain serialized Marauroa objects. |
|||
20:49kiheruusing the binary serialization? |
|||
20:50hendrikYes :-/ |
|||
20:51hendrik"typed arrays javascript" is something to look into. |
|||
20:51kiheruouch |
|||
20:51hendrikI am not sure if that will help. |
|||
20:52hendrikWe could blame it on Stendhal. But perhaps there is a way to solve the issue in Marauroa. I am not sure, yet. |
|||
20:52kiheruat which point is the serialization format decided? I mean most of the traffic gets serialized to json, but apparently some using the binary |
|||
20:53kiheru(I should remember what map_data does, but apparently I don't) |
|||
20:53hendrikIt is decided when a Message is sent to the client. The subclasses of Message have both methods. |
|||
20:53hendrikSo a json message with the transfer data is sent to the client. This part works. |
|||
20:54hendrikBut the data that Stendhal tells Marauroa to transfer, is already binary serialized. |
|||
20:55hendrikpublic void transferContent(RPObject target, List<TransferContent> content) { |
|||
20:55hendrikin RPServerManager. |
|||
20:56hendrikI still need to look at the Stendhal side. |
|||
20:58hendrikLet's postpone that until after the meeting. |
|||
20:59hendrikAnother open issue is, that I deleted the Maven pom.xml file. |
|||
20:59hendrikI was unable to get travis pick up "ant test" with the pom.xml file existing. And "maven test" is not working. |
|||
21:00hendrikWhile I don't think that Maven support is a blocker, it would be nice to restore it. |
|||
21:00hendrikAnd then there is the breaking change of gameEvents.source (String) to gameEvents.character_id (int). |
|||
21:00markusI could look into that. :) |
|||
21:01hendrikthank you. |
|||
21:01hendrikWhile I prefer to use references, I am not sure about changing the table structure before 4.0 |
|||
21:02markusIf we want to release soon we should consider delaying that one. Although I think it is a good idea. |
|||
21:02hendrikHaving a new major version number is kind of a justification for such changes. |
|||
21:02hendrikBut, ... what Markus just said. |
|||
21:04hendrikDatabasewise, I would automatically rename the existing gameEvents table and create a new one with the new structure, setting the id to the last id from the old table. |
|||
21:05hendrikAs updating the data will likely take a couple of days. |
|||
21:05postmanarianne: balaur * developers #274 [Marauroa] Maven for Marauroa http://sf.net/p/arianne/developers/274 |
|||
21:06hendrikBut there are cases in which source is not refering to player. For example on kill events, source refers to the killer, which may be a creature. |
|||
21:09hendrikbalaur raised some other important points about Maven in the linked ticket. |
|||
21:09markusyes, saw that :) |
|||
21:10hendrikAnything else we need to keep in mind for the Marauroa release? |
|||
21:10markusmaking character_id nullable is no option? |
|||
21:11hendrikAnd have that in addition to source? |
|||
21:13hendrikThis may need some more thinking. I tend to exclude it from 4.0. |
|||
21:13hendrikOpinions about including or excluding? |
|||
21:14markusI'd go for excluding. |
|||
21:14markusMight be an option to make that an addional column. |
|||
21:16hendrikOkay, so let's consider that change not a release blocker. |
|||
--> |
|||
== Doing Sendhal releases == |
== Doing Sendhal releases == |
||
| Line 214: | Line 92: | ||
== PostgreSQL support == |
== PostgreSQL support == |
||
* Long term possible objective using JSON datatype for rpobject.object |
|||
* ... |
|||
* Adding PostgreSQL support to Marauroa is little work |
|||
<!-- |
|||
* Hallo of fame script and website need to support Postgres, too. |
|||
21:51hendrik4. Postgres support |
|||
* Website |
|||
21:51hendrikMarkus? |
|||
** Website currently uses deprecated old mysql api. It needs to be rewritten using [https://php.net/manual/de/book.pdo.php PDO] anyway to be compatible with the next PHP version |
|||
21:52markusI just had the idea of starting to support postgres. |
|||
** Old php mysql code and new PDO code will result in doubling the number of database connectings, and having isolated transactions. |
|||
21:52markusOne thing postgres supports is a JSON datatype for columns which could serve as replacement for the blobs. |
|||
** We can do that in (rather large) chunks by converting all the code that goes to the wiki-database first. Then the website database, and finally the game-database. |
|||
21:54hendrikI think adding Postgres support to Marauroa as such is quite simple and little work. |
|||
21:54balaurThat would be nice |
|||
21:54markusJSON datatype has one advantage to a blob: postgres can issue queries on JSON columns. for example: all json objects with attribute a="blah" |
|||
21:54hendrikDoes it support complex nested objects? |
|||
21:54markusI agree on that part, hendrik. |
|||
21:55hendrikI just realized, that adding postgres support to the stendhal website will be a lot of work. But we have to do that work anyway sooner or later. |
|||
21:56hendrikThe website uses deprecated mysql specific php libraries that have been removed in the next php version. |
|||
21:56markusthe documentation says, it supports RFC 4627 http://www.postgresql.org/docs/9.3/static/datatype-json.html |
|||
21:57markusI think a good first step would be to add plain postgres support before considering JSON |
|||
21:57hendrikYes, I agree. |
|||
21:58hendrikSo that is: Marauroa, Stendhal, Hall of fame script, website. |
|||
21:58hendrikI can do the Marauroa part soon. |
|||
22:00markusI could look into Stendhal to see what's needed there. |
|||
22:01hendrikI think mixing old php mysql code and new php db code will result in doubling the number of database connectings, and having isolated transactions. |
|||
22:02hendrikSo that will be a project for a branch. Luckily we have git nowadays. |
|||
22:03hendrikWe can do that in (rather large) chunks by converting all the code that goes to the wiki-database, game-database, website-date. |
|||
--> |
|||
== Other topics == |
|||
* ... |
|||
Note: We may want to postpone topics regarding Stendhal features to a follow up meeting. But it is okay to list them here. This will help with preparing an agenda for the next meeting. |
|||
Latest revision as of 07:19, 12 January 2016
We will meet in #arianne on irc.freenode.net, https://webchat.freenode.net/?channels=arianne on Wednesday, 6th January, 2016 at 20.00 server time (19.00 GMT)
Sourceforge, Github, Travis
In August after an extended downtime of Sourceforge, we created the Arianne organisation on Github and mirrored the Stendhal git repository. Back than the mirroring was only from Sourceforge to github, and it was done by the testserver build script once a night.
Since then we have improved the situation significantly: The mirroring is now done in both directions, and it is done almost instantaneously. This means it does not matter anymore whether commits are done to Sourceforge or Github. So we can use Github features such as pull request. And we can make use of Travis for running the JUnit test. Two way git mirror explains the details.
In addition to Stendhal, Marauroa is now on github, too.
All JUnit tests for Marauroa [1] and Stendhal [2] are passing. So before we do a new release, we can just check there. In addition, Travis will execute the test suite for each pull request and post the result directly to the pull request page. Unfortunetelly there is nice output of failed tests. So in case of failures, we have to look at the console log or execute the tests locally.
Failed tests are reported to IRC again. We should add the build-passing image and Github to https://arianne-project.org and Stendhal Testing.
We should document pull requests on How To Create Patch For Stendhal. We can recommend pull requests, but accept patches as well. For those familiar with Github pull requests are easy. But patches are simpler for the others.
Prepaing release of Marauroa 4.0
A couple of years ago we started to experiment with web clients. We did several attempts using different technologies. To cut a long story short: Only recent browser developments enabled us to build a working webclient. All this time, most of Marauroa's development was done on a branch named perception_json, with only minor changes being backported to master.
Without a real working client, I was afraid that the Marauroa interface were not tested against real requirement. But the last attempt to write a Stendhal webclient is very promissing (although there is still a lot of work required). So perception_json branch was merged into master recently.
We should do a Marauroa release in the near future: Release bockers are tagged with MARAUROA_03_99
- Most important is updating documentation and examples.
- Stendhal should be able load the maps from transfer-content so that that part of the json protocol gets tested as well. Currently Stendhal uses Marauroa's binary serialization internally to provide content. So either the JavaScript needs to understand it, or the server needs to provide alternative content
- Changing structure of gameEvents table to use references instead of source string. Postponed after 4.0
- Re-adding support for Marven which was deleted because mvn test is broken and Travis insists on using Maven instead of Ant. We should aim to restore it before release. But this is not a release blocker.
Doing Sendhal releases
- At the moment I am the only person who can do a Stendhal release. If I am not available and there is an active contributor, lots of changes pill up. This is hard to review, and I assume it may be frustrating for contributors. Creating a release, however, is both complicated and time consuming.
- ...
PostgreSQL support
- Long term possible objective using JSON datatype for rpobject.object
- Adding PostgreSQL support to Marauroa is little work
- Hallo of fame script and website need to support Postgres, too.
- Website
- Website currently uses deprecated old mysql api. It needs to be rewritten using PDO anyway to be compatible with the next PHP version
- Old php mysql code and new PDO code will result in doubling the number of database connectings, and having isolated transactions.
- We can do that in (rather large) chunks by converting all the code that goes to the wiki-database first. Then the website database, and finally the game-database.