Opened 5 years ago

Last modified 2 weeks ago

#2835 released enhancement (fixed)

Google AppEngine compatibility

Reported by: Joonas Lehtinen Owned by: Marc Englund
Priority: critical Milestone: Vaadin 6.1.0
Component: Core Framework Version: 5.3.1
Keywords: Cc:
Depends on: #695
Workaround:
Verified:
Fv: no Pro Account: Vote for Feature

Description (last modified by Artur Signell)

Known issues (resolved):

  • Toolkit is not serializable #695
  • AppEngine wants to serialize HttpSession after each request - even if the serialization would work, this would probably be a performance problem
  • Locale.setDefault() is not allowed (#2840)
  • Force AppEngine to re-serialize after transaction - otherwise it will de-serialize an old version of the object and thus server state does not change.
  • #3015 class Object instances (used in IndexedContainer and Form as identifiers) are not serializable
  • #3057 DateField deserialization does not work
  • #3064 GAE cannot handle primitive class references

(http://code.google.com/p/googleappengine/issues/detail?id=1700)

Known issues (TODO):

When done, please update http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

Attachments (1)

calc-test.zip (99.9 KB) - added by Joonas Lehtinen 5 years ago.
Detailed serialization logs for Calc to demonstrate session state problem

Download all attachments as: .zip

Change History (45)

comment:1 Changed 5 years ago by Artur Signell

  • Description modified (diff)

comment:2 follow-up: Changed 5 years ago by Joonas Lehtinen

Benchmarked serialization. Serializing Sampler takes 16ms on my laptop (and produces some 170kb of serialized session data). Thus serialization might not be that big performance problem after all - at least for some applications.

comment:3 in reply to: ↑ 2 Changed 5 years ago by Joonas Lehtinen

See [7388] for serialization time measurement

comment:4 Changed 5 years ago by Joonas Lehtinen

Experimenting on http://tkgaetest.appspot.com/

What works this far:

  • Deployment of demos to App Engine
  • Starting Hello World, Calculator, Sampler and AddressBook
  • Sessions are stored on Google DataStore (for example - running calculator grows session to 11208 bytes)

What does not work:

  • For some reason state does not fully persist: *for example, calculator memory isresetted between requests

comment:5 Changed 5 years ago by Joonas Lehtinen

  • Priority changed from major to critical
  • Summary changed from Add Google AppEngine compatibility to Google AppEngine compatibility

comment:6 Changed 5 years ago by Joonas Lehtinen

  • Description modified (diff)

comment:7 Changed 5 years ago by Joonas Lehtinen

Here is a bit more detailed description of the "session clear" -problem:

For the following application:

package com.itmill.toolkit.demo;

import java.io.Serializable;
import java.util.Iterator;

import com.itmill.toolkit.Application;
import com.itmill.toolkit.service.ApplicationContext.TransactionListener;
import com.itmill.toolkit.ui.Button;
import com.itmill.toolkit.ui.GridLayout;
import com.itmill.toolkit.ui.Label;
import com.itmill.toolkit.ui.Window;

public class Calc extends com.itmill.toolkit.Application implements
        Serializable, TransactionListener {

    private static final long serialVersionUID = -2185106514237814051L;

    private double current = 0.0;
    private double stored = 0.0;
    private char lastOperationRequested = 'C';

    // User interface components
    private final Label display = new Label("0.0");
    private final GridLayout layout = new GridLayout(4, 5);

    // Application initialization creates UI and connects it to business logic
    @Override
    public void init() {

        // Place the layout to the browser main window
        setMainWindow(new Window("Calculator Application", layout));

        // Create and add the components to the layout
        layout.addComponent(display, 0, 0, 3, 0);
        for (String caption : new String[] { "7", "8", "9", "/", "4", "5", "6",
                "*", "1", "2", "3", "-", "0", "=", "C", "+" }) {
            Button button = new Button(caption, new Button.ClickListener() {
                public void buttonClick(Button.ClickEvent event) {

                    // On button click, calculate and show the result
                    display.setValue(calculate(event.getButton()));
                }
            });
            layout.addComponent(button);
        }

        getContext().addTransactionListener(this);
    }

    // Calculator "business logic" implemented here to keep the example minimal
    private double calculate(Button buttonClicked) {
        char requestedOperation = buttonClicked.getCaption().charAt(0);
        if ('0' <= requestedOperation && requestedOperation <= '9') {
            current = current * 10
                    + Double.parseDouble("" + requestedOperation);
            return current;
        }
        switch (lastOperationRequested) {
        case '+':
            stored += current;
            break;
        case '-':
            stored -= current;
            break;
        case '/':
            stored /= current;
            break;
        case '*':
            stored *= current;
            break;
        case 'C':
            stored = current;
            break;
        }
        lastOperationRequested = requestedOperation;
        current = 0.0;
        if (requestedOperation == 'C') {
            stored = 0.0;
        }
        return stored;
    }

    private int transactionCount = 0;
    private final long id = System.currentTimeMillis();

    public void transactionEnd(Application application, Object transactionData) {
        System.err.println("Calc: End transaction " + transactionCount
                + ": id=" + id + " current=" + current + " display=" + display);
        System.err.print("Apps: ");
        for (Iterator<Application> i = getContext().getApplications()
                .iterator(); i.hasNext();) {
            Application a = i.next();
            System.err.print(a.getClass());
            if (a instanceof Calc) {
                System.err.print("(id=" + id + " display=" + ((Calc) a).display
                        + ")");
            }
            if (i.hasNext()) {
                System.err.print(",");
            }
        }
        System.err.println("");
    }

    public void transactionStart(Application application, Object transactionData) {
        transactionCount++;
        System.err.println("Calc: Start transaction " + transactionCount
                + ": id=" + id + " current=" + current + " display=" + display);
    }
}

We get this LOG:

  • Pressed key 2 - note that somehow the application state (current) has been cleared, but NOT application id !?!?
    • 04-12 04:44AM 24.501 [tkgaetest/1.332735553755014911].<stderr>: Calc: Start transaction 3: id=1239536655081 current=0.0 display=0.0
    • 04-12 04:44AM 24.503 [tkgaetest/1.332735553755014911].<stderr>: Calc: End transaction 3: id=1239536655081 current=2.0 display=2.0
    • 04-12 04:44AM 24.503 [tkgaetest/1.332735553755014911].<stderr>: Apps: class com.itmill.toolkit.demo.Calc(id=1239536655081 display=2.0)
  • Pressed key 1
    • 04-12 04:44AM 21.777 [tkgaetest/1.332735553755014911].<stderr>: Calc: Start transaction 3: id=1239536655081 current=0.0 display=0.0
    • 04-12 04:44AM 21.788 [tkgaetest/1.332735553755014911].<stderr>: Calc: End transaction 3: id=1239536655081 current=1.0 display=1.0
    • 04-12 04:44AM 21.788 [tkgaetest/1.332735553755014911].<stderr>: Apps: class com.itmill.toolkit.demo.Calc(id=1239536655081 display=1.0)
  • What is this?
    • 04-12 04:44AM 18.029 [tkgaetest/1.332735553755014911].<stderr>: Calc: Start transaction 2: id=1239536655081 current=0.0 display=0.0
    • W 04-12 04:44AM 18.098 [tkgaetest/1.332735553755014911].<stderr>: Calc: End transaction 2: id=1239536655081 current=0.0 display=0.0
    • 04-12 04:44AM 18.098 [tkgaetest/1.332735553755014911].<stderr>: Apps: class com.itmill.toolkit.demo.Calc(id=1239536655081 display=0.0)
  • This is probably the first UIDL fetch from the application with repaintAll
    • 04-12 04:44AM 15.980 [tkgaetest/1.332735553755014911].<stderr>: Calc: Start transaction 2: id=1239536655081 current=0.0 display=0.0
    • 04-12 04:44AM 15.981 [tkgaetest/1.332735553755014911].<stderr>: Calc: End transaction 2: id=1239536655081 current=0.0 display=0.0
    • 04-12 04:44AM 15.981 [tkgaetest/1.332735553755014911].<stderr>: Apps: class com.itmill.toolkit.demo.Calc(id=1239536655081 display=0.0)
  • Initial fetch of the application http://tkgaetest.appspot.com/calc/ - app created
    • 04-12 04:44AM 15.264 [tkgaetest/1.332735553755014911].<stderr>: Calc: Start transaction 1: id=1239536655081 current=0.0 display=0.0
    • 04-12 04:44AM 15.346 [tkgaetest/1.332735553755014911].<stderr>: Calc: End transaction 1: id=1239536655081 current=0.0 display=0.0
    • 04-12 04:44AM 15.347 [tkgaetest/1.332735553755014911].<stderr>: Apps: class com.itmill.toolkit.demo.Calc(id=1239536655081 display=0.0)

comment:8 Changed 5 years ago by Joonas Lehtinen

  • Description modified (diff)

Changed 5 years ago by Joonas Lehtinen

Detailed serialization logs for Calc to demonstrate session state problem

comment:9 Changed 5 years ago by Joonas Lehtinen

Attached another test that demonstrates the problem even better. Some findings:

  • Google App Engine always de-serializes the session in the beginning of a transaction.
    • This is bad for performance
  • App Engine does not re-serialize after transaction.

comment:10 Changed 5 years ago by Joonas Lehtinen

  • Description modified (diff)

comment:11 Changed 5 years ago by Joonas Lehtinen

Just a consideration - maybe this problem has something to do with clone(), hashCode() or equals()? Maybe Google does not re-serialize session if the hashcode did not change?

Fairly simple idea of implementing these would be using serialization (to bytearray) to implement these. Heavy, but correct.

comment:12 Changed 5 years ago by Artur Signell

Seems that GAE does not serialize the session if it has not been changed. It does not seem to check anything inside the session(?), at least WebApplicationContext hashCode/equals are not called. Possibly using listeners attached to the HttpSession?

Touching the session after each request ensures it is re-serialized. Added a workaround in [7391](6.0). The correct way to do it should still be investigated.

comment:13 Changed 5 years ago by Joonas Lehtinen

Artur - is there anything missing still, or is this ticket fixed by [7389], [7387] and [7391]?

comment:14 Changed 5 years ago by Artur Signell

  • Owner changed from ticketmaster to Artur Signell
  • Status changed from new to accepted

Sampler can be found at http://itmill-sampler.appspot.com/
There seems to still be some problems for instance when opening a Select in the Form sample -> Out of Sync

comment:15 Changed 5 years ago by Joonas Lehtinen

Lot of out-of-sync errors for me (for the url you provided above).. It seems that I can navigate to a feature, but any changes there causes out-of-sync.

comment:16 Changed 5 years ago by Artur Signell

Also a problem is that every image request (application resource) apparently causes a session deserialization.

comment:17 Changed 5 years ago by Joonas Lehtinen

This sounds natural as the application resource comes from the application and application lives in session...

One huge problem i see is that our synchronization blocks wont work. Parallel UIDL or application resources could cause really strange things...

Any ideas?

comment:18 Changed 5 years ago by Joonas Lehtinen

Google really really should consider keeping sessions live for a while before serializing them. This would make App Engine perform a lot better and would prevent this kind of problems from happening..

comment:19 Changed 5 years ago by Artur Signell

  • Description modified (diff)

Added #2842 to the list of known issues

comment:20 Changed 5 years ago by Joonas Lehtinen

Is there a definitive list of classes that does not work in AppEngine?

comment:21 Changed 5 years ago by Henri Sara

  • Description modified (diff)

Added known issue #3015.

comment:22 Changed 5 years ago by Artur Signell

  • Description modified (diff)

comment:24 Changed 5 years ago by Artur Signell

  • Description modified (diff)

worked around #3057

comment:25 Changed 5 years ago by Artur Signell

  • Description modified (diff)

comment:26 Changed 5 years ago by Artur Signell

  • Description modified (diff)

comment:27 Changed 5 years ago by Artur Signell

Sampler seems to work correctly in GAE now: http://itmill-sampler.appspot.com

comment:28 Changed 5 years ago by Joonas Lehtinen

Wow... Works, but GAE is slow as hell in loading those icons...

comment:29 Changed 5 years ago by Joonas Lehtinen

Hmmm.. Getting a lot of out of sync errors. Maybe related to running app in ff and safari at the same time (as well as twiiting the url)... Can it really handle over 1 users?

comment:30 Changed 5 years ago by Joonas Lehtinen

At least this sample can handle multiple browsers:
http://vaadin-addressbook-test.appspot.com/

Maybe sync errors have something to do with loading tens of app resources (icons) in parallel...

comment:31 Changed 5 years ago by Marc Englund

  • Milestone changed from IT Mill Sponsored Backlog to Vaadin 6.1.0
  • Owner changed from Artur Signell to Marc Englund

comment:32 Changed 5 years ago by Marc Englund

Committed a version that attempts to synchronize via memcache in [8614]

This version still uses GAE sessions; it has a number of limitations because of this, and still a potential race-condition (depending on G's implementataion - I've been unable to cause it thus far).

It's a big improvement from previous versions that caused race-conditions all the time.

Still, it looks like handing the serialization our selves, and forgetting GAEs sessions, is the only way to make this _good_. This will require significantly more changes, though.

comment:33 Changed 5 years ago by Joonas Lehtinen

Marc, could you elaborate those mentioned limitations a bit here as many people considering to use GAE for Vaadin have been referred to this ticket for status.

comment:34 Changed 5 years ago by Marc Englund

Yes, I'll write more details asap. The current status in short, however, is this: the current trunk will work much better than previously, but is not perfect - we're planning to do all the session handling our selves in order to do this properly. I'll start looking into this tomorrow, and by Thursday I should know whether or not it's feasible to include that in the next release.

comment:35 Changed 5 years ago by Marc Englund

Well, an intermediate update: the we'll-handle-everything-ourselves version is progressing nicely, so IMO there is no point going trough the exact limitations of the versions that tries to use GAE sessions. Basically we're only missing a cleanup robot and testing, testing, testing.

comment:36 Changed 5 years ago by Joonas Lehtinen

Great news. Will this be in 6.1.0?

comment:37 Changed 5 years ago by Artur Signell

The own session handling is included in 6.1.0-pre2.

comment:38 Changed 5 years ago by Marc Englund

Fixed in [8669]
Some serialization bugs fixed in [8705] (not actually synchronization related, but Sampler did not work on GAE and could not be tested before fixing)

Current notes and limitations for Google Appengine

  • applications must use GAEApplicationServlet instead of ApplicationServlet (web.xml)
  • session support must be enabled in appengine-web.xml:
    <sessions-enabled>true</sessions-enabled>
    
  • avoid using the session for storage; usual appengine limitations apply (no synchronization, i.e unreliable)
  • Vaadin uses memcache for mutex, the key is of the form _vmutex<sessionid>
  • the Vaadin WebApplicationContext is serialized separately into memcache and datastore; the memcache key is _vac<sessionid> and the datastore entity kind is _vac with ids of the type _vac<sessionid>
  • DO NOT update application state when serving a ApplicationResource (e.g ClassResource.getStream())
  • AVOID (or be very careful when) updating application state in a transactionListener - it's called even when the application is not locked and won't be serialized (e.g ApplicationResource), and changes can thus go missing (it should be safe to update things that can be safely discarded later - i.e valid only for the current request)
  • the application remains locked during uploads - no progressbar possible

comment:39 Changed 5 years ago by Marc Englund

  • Resolution set to fixed
  • Status changed from accepted to closed

Error notifications made overrideable in [8710], closing.

comment:40 Changed 5 years ago by Marc Englund

  • Resolution fixed deleted
  • Status changed from closed to reopened

...adn I closed the wrong ticket - we have too many for GAE :-)

comment:41 Changed 5 years ago by Joonas Lehtinen

What is still in TODO? Should this ticket be closed now when 6.1 with GAE compatibility has been released? If there is still some issue - it could be clearer to report that separately....

(keeping ticket open gives impression that vaadin is not gae compatible)

comment:42 Changed 5 years ago by Artur Signell

  • Milestone changed from Vaadin 6.1.1 to Vaadin 6.1.0
  • Resolution set to fixed
  • Status changed from reopened to closed

Created #3320 (Create a session cleaner for GAE) and #3321 (Implement session splitting for GAE) for remaining issues.

comment:43 Changed 4 months ago by Artur Signell

  • Fv unset

comment:44 Changed 2 weeks ago by Artur Signell

  • Status changed from closed to released
Note: See TracTickets for help on using tickets.